← American Express Interview Insights

American Express·AI Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Amex for an AI Engineer role, focused almost entirely on fault tolerance and reliability for AI systems. Pretty deep technically, more distributed systems than ML, which I wasn't fully expecting.

Questions Asked (6)

Q1

How would you design a fault-tolerant AI system that can recover from failures? Walk through your approach to stateless versus stateful components and how you'd handle persistent state.

System DesignTechnical Trade-offs
Author's notes

This was the main question and it sprawled for like 40 minutes.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the system's requirements and constraints, then propose a layered architecture that separates stateless and stateful components. Explain how you would use replication, checkpointing, and idempotent operations to achieve fault tolerance, and discuss trade-offs between consistency, availability, and latency. Finally, walk through a concrete failure scenario to demonstrate your design's resilience.

Pro tip: Emphasize idempotency and exactly-once processing for stateful operations, as these are critical in financial systems like American Express where data integrity is paramount. Also, mention how you would monitor and test fault tolerance through chaos engineering.

1. Clarify Requirements and Constraints

Ask about expected scale, latency requirements, consistency needs, and recovery time objectives (RTO/RPO). This shows you understand that fault tolerance design is context-dependent.

2. Design Stateless Components for Horizontal Scalability

Explain that stateless components (e.g., API gateways, model inference servers) can be easily replicated and load-balanced, with failures handled by health checks and automatic restarts.

3. Design Stateful Components with Replication and Checkpointing

Describe how stateful components (e.g., feature stores, model training jobs) use replication (e.g., primary-backup, quorum) and periodic checkpointing to recover from failures.

4. Handle Persistent State with Durable Storage and Idempotent Operations

Discuss using distributed databases (e.g., Cassandra, DynamoDB) with replication, and ensuring operations are idempotent to avoid duplicate processing during retries.

5. Implement Monitoring, Testing, and Recovery Procedures

Mention proactive monitoring, automated failover, and chaos engineering to validate fault tolerance. Also, outline a recovery playbook for different failure scenarios.

Key Points to Mention

  • Stateless vs. stateful components: stateless for scalability, stateful for persistence with replication.
  • Replication strategies: primary-backup, multi-leader, quorum-based for stateful components.
  • Checkpointing and write-ahead logging for model training and inference state.
  • Idempotency and exactly-once semantics to handle retries safely.
  • Trade-offs: consistency vs. availability (CAP theorem), latency vs. durability.
  • Monitoring, alerting, and chaos engineering to ensure fault tolerance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How do you ensure idempotent request handling when retrying LLM or tool calls, and when do you actually need exactly-once semantics?

System DesignTechnical Trade-offs
Author's notes

Answered this mostly fine.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining idempotency in the context of LLM and tool calls, emphasizing the need for unique request IDs and deduplication. Then discuss practical techniques like idempotency keys, state tracking, and transactional boundaries. Finally, explain when exactly-once semantics are necessary versus at-least-once with idempotency, considering business impact and cost.

Pro tip: Highlight that exactly-once is often overkill; most systems achieve effectively-once by combining at-least-once delivery with idempotent operations. Mention that in financial systems like American Express, auditability and reconciliation are key, so logging every attempt with idempotency keys is crucial.

1. Define Idempotency and Its Importance

Explain that idempotency ensures repeated requests have the same effect as a single request, preventing duplicate side effects like double charges or duplicate tool executions.

2. Implement Idempotency Mechanisms

Describe using unique idempotency keys per logical request, storing them with request state (e.g., in a database or cache) to detect and ignore duplicates. For LLM calls, include a request ID in the prompt or metadata and have the model echo it back.

3. Handle Retries and State Management

Outline a retry strategy with exponential backoff and jitter, and ensure that the state of the operation (e.g., pending, completed) is persisted so that retries can check status before re-executing.

4. Determine When Exactly-Once Is Needed

Discuss scenarios where exactly-once semantics are critical, such as financial transactions or irreversible actions, and where at-least-once with idempotency suffices, like non-critical logging or analytics.

5. Balance Trade-offs and Costs

Explain the trade-offs: exactly-once often requires distributed transactions or complex coordination, which can impact latency and availability. Idempotency with at-least-once is simpler and more scalable.

Key Points to Mention

  • Idempotency keys: unique identifiers for each logical request to deduplicate retries.
  • State tracking: persisting request status (e.g., in a database) to avoid re-execution.
  • Retry strategies: exponential backoff with jitter to handle transient failures.
  • Exactly-once vs. at-least-once: exactly-once is needed for critical, non-idempotent operations; otherwise, at-least-once with idempotency is sufficient.
  • LLM-specific challenges: non-deterministic outputs and potential side effects from tool calls, requiring careful idempotency design.
  • Financial context: audit trails and reconciliation are essential; log all attempts with idempotency keys for compliance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What's your strategy for retry logic and circuit breakers specifically around LLM API calls and external tool integrations?

System DesignAPI & Integrations
Author's notes

Pretty comfortable here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by distinguishing between retryable and non-retryable errors for LLM APIs and external tools, then describe a layered strategy with exponential backoff and jitter for transient failures. Explain how circuit breakers prevent cascading failures by isolating unhealthy dependencies, and emphasize the need for fallbacks and observability in a regulated environment like American Express.

Pro tip: Mention that LLM calls are expensive and non-deterministic, so you should avoid retrying on content-related errors (e.g., safety filters) and instead retry only on infrastructure errors like timeouts or rate limits. Also, highlight that circuit breakers should be per-endpoint or per-tool, not global, to avoid unnecessary outages.

1. Classify errors and define retry policies

Identify which errors are transient (e.g., 429, 503, timeouts) and which are permanent (e.g., 400, 401, content policy violations). Set retry policies only for transient errors, with a maximum retry count and total timeout.

2. Implement exponential backoff with jitter

Use exponential backoff with randomized jitter to avoid thundering herd problems. For LLM APIs, consider longer base delays due to rate limits and higher latency.

3. Add circuit breakers per dependency

Wrap each external tool or LLM endpoint with a circuit breaker that opens after a threshold of failures, preventing repeated calls to a failing service. Use half-open state to test recovery.

4. Design fallbacks and graceful degradation

When a circuit is open or retries are exhausted, fall back to a cached response, a simpler model, or a default behavior. Ensure the user experience degrades gracefully without exposing errors.

5. Monitor and tune with observability

Track retry rates, circuit breaker state changes, and latency. Use metrics and logs to tune thresholds and backoff parameters, and set up alerts for abnormal patterns.

Key Points to Mention

  • Exponential backoff with jitter to avoid synchronized retries
  • Circuit breaker states: closed, open, half-open, and per-endpoint isolation
  • Idempotency keys for safe retries on non-idempotent operations
  • Fallback strategies: cached responses, alternative models, or degraded functionality
  • Observability: metrics, logging, and alerting on retry and circuit breaker events
  • Compliance and cost considerations: avoid retrying on content policy violations and limit retries to control LLM API costs

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you design graceful degradation when a downstream model or retrieval backend becomes unhealthy?

System DesignTechnical Trade-offs
Author's notes

I talked about fallback chains, cached responses, and surfacing degraded-mode signals to users.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining what 'unhealthy' means for each downstream component and the acceptable degradation levels. Then, propose a layered fallback strategy that includes caching, circuit breakers, and alternative models, while ensuring observability and graceful user experience. Finally, discuss trade-offs between consistency, latency, and cost.

Pro tip: Emphasize the importance of monitoring and alerting on degradation metrics, and consider implementing a 'degraded mode' that is explicitly communicated to users to manage expectations.

1. Define Health and Failure Modes

Identify what constitutes an unhealthy state for each downstream component (e.g., high latency, error rates, timeouts) and the potential failure modes.

2. Design Fallback Layers

Propose multiple fallback options such as cached responses, simpler models, rule-based systems, or static content, ordered by quality and cost.

3. Implement Circuit Breakers and Timeouts

Use circuit breakers to detect failures and prevent cascading issues, and set appropriate timeouts to avoid hanging requests.

4. Ensure Observability and Feedback

Instrument the system to log degradation events, monitor key metrics, and alert on thresholds to enable quick response.

5. Communicate and Iterate

Define how to inform users or downstream systems about degraded service, and establish a process to review and improve the degradation strategy.

Key Points to Mention

  • Circuit breaker pattern and its role in preventing cascading failures
  • Caching strategies (e.g., Redis, in-memory) for frequently accessed data
  • Fallback to simpler models or rule-based systems when primary model fails
  • Timeouts and retries with exponential backoff
  • Monitoring and alerting on health metrics (latency, error rates, saturation)
  • Trade-offs between consistency, availability, and latency (CAP theorem)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How do you approach multi-region replication and failover for AI systems, including both model artifacts and runtime state?

System DesignTechnical Trade-offs
Author's notes

Harder than it sounds because model artifacts are big and runtime state for things like conversation history has consistency requirements.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by distinguishing between model artifacts (static, versioned) and runtime state (dynamic, session-specific), then outline a multi-region strategy that ensures low-latency inference, high availability, and consistency where needed. Emphasize trade-offs between consistency, latency, and cost, and tie your approach to American Express's needs for reliability, compliance, and global scale.

Pro tip: Highlight the importance of idempotent failover and graceful degradation—showing you understand that not all state needs strong consistency and that partial failures should not cascade. Mention that you'd validate failover with chaos engineering and game days, which demonstrates operational maturity.

1. Clarify requirements and constraints

Ask about latency SLAs, consistency requirements, data residency, and budget. Distinguish between model artifacts (read-only, versioned) and runtime state (user sessions, feature stores, caches).

2. Design artifact replication

Use a globally distributed artifact registry (e.g., S3 cross-region replication, container registries) with immutable versioning. Ensure models are pre-warmed in each region to avoid cold starts during failover.

3. Design runtime state replication

Classify state by consistency needs: session state may use eventual consistency with regional caches, while critical state (e.g., transaction context) may require synchronous replication or a globally consistent store. Use change data capture and conflict-free replicated data types where appropriate.

4. Implement failover and routing

Use global load balancing with health checks and automated DNS failover. Define failover policies: active-active for stateless inference, active-passive for stateful components. Ensure idempotent retries and circuit breakers to prevent cascading failures.

5. Validate and monitor

Test failover regularly with chaos engineering and game days. Monitor replication lag, error rates, and latency. Implement canary deployments for model updates across regions.

Key Points to Mention

  • Distinction between model artifacts (static, versioned) and runtime state (dynamic, session-specific)
  • Trade-offs: consistency vs. latency vs. cost (e.g., synchronous vs. asynchronous replication)
  • Use of global load balancing, health checks, and automated failover
  • Idempotency and graceful degradation to avoid cascading failures
  • Data residency and compliance considerations (e.g., GDPR, financial regulations)
  • Testing failover with chaos engineering and game days

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How do you handle prompt and tool versioning to maintain reproducibility across different model versions in a production system?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second on this one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining reproducibility as a first-class requirement and describe a versioning strategy that treats prompts, tools, and models as immutable artifacts with explicit dependencies. Then walk through a concrete system design that ties these versions together via a manifest or registry, and discuss trade-offs like backward compatibility, testing, and rollback.

Pro tip: Emphasize that reproducibility isn't just about version numbers—it's about capturing the full environment (model version, prompt template, tool definitions, and even decoding parameters) in a single, immutable artifact. Mention that you'd enforce this via CI/CD and automated regression tests to catch drift early.

1. Define versioning granularity

Decide what constitutes a version for prompts, tools, and models. For prompts, use semantic versioning with hashes; for tools, version APIs and schemas; for models, pin to specific model IDs and provider versions.

2. Create a manifest or registry

Store all versioned components in a central registry (e.g., Git, MLflow, or a custom service) and generate a manifest that locks the exact versions used for each deployment or experiment.

3. Integrate with CI/CD and testing

Automate validation that the manifest is consistent and that changes to any component trigger regression tests against a golden dataset to detect behavioral drift.

4. Handle model upgrades and backward compatibility

When upgrading models, run A/B tests or shadow deployments to compare outputs, and maintain compatibility layers or fallbacks for older prompts/tools if needed.

5. Monitor and rollback

Log the manifest version with each inference request, monitor for anomalies, and enable quick rollback to a previous manifest if issues arise.

Key Points to Mention

  • Immutable artifacts and semantic versioning for prompts and tools
  • Centralized registry and manifest for dependency locking
  • Automated regression testing and CI/CD integration
  • Model version pinning and provider-specific versioning
  • Backward compatibility and deprecation strategies
  • Observability: logging versions with requests and rollback mechanisms

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.