← American Express Interview Insights

American Express·AI Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Amex for an AI Engineer role, focused entirely on long-running multi-agent systems. Pretty deep cut for a single question session. Left feeling like I'd covered maybe 60% of what they were actually looking for.

Questions Asked (1)

Q1

How would you design and operate a multi-agent system that runs for hours or days, potentially with agents cooperating with each other? Walk through how you'd handle durable state, crash recovery, idempotency, retries, supervision, inter-agent communication, observability, cost management, and human-in-the-loop interrupts. Also cover failure modes like drift, looping, and runaway costs.

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

This is basically a full distributed systems design question dressed up in AI clothing, and I didn't fully register that until I was halfway through rambling about checkpointing.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a distributed systems challenge, emphasizing durability, idempotency, and observability. Then walk through the architecture in layers: state management, communication, supervision, and cost controls. Finally, discuss failure modes and human-in-the-loop as first-class concerns, tying back to American Express's need for reliability and compliance.

Pro tip: Emphasize that you'd design for failure from day one, using patterns like idempotent operations, checkpointing, and circuit breakers. Mention that you'd instrument everything with metrics and traces to detect drift and loops early, and set hard budget caps to prevent runaway costs.

1. Define the system architecture and state model

Outline the multi-agent architecture: roles, communication patterns (e.g., message queues, blackboard), and durable state storage (e.g., database, event log). Specify how state is checkpointed and recovered after crashes.

2. Ensure reliability and idempotency

Describe how you'd make operations idempotent using unique IDs and deduplication. Explain retry policies with exponential backoff and jitter, and how supervision trees or orchestrators handle agent failures.

3. Implement observability and cost controls

Detail logging, metrics, and tracing for each agent and interaction. Set up cost monitoring with budgets, alerts, and automatic throttling or shutdown when thresholds are exceeded.

4. Handle failure modes and human-in-the-loop

Discuss detection and mitigation of drift (e.g., periodic re-evaluation), looping (e.g., cycle detection), and runaway costs. Explain how humans can interrupt, approve, or correct agent actions via APIs or dashboards.

5. Summarize trade-offs and operational maturity

Conclude by highlighting key trade-offs (e.g., consistency vs. availability, cost vs. latency) and how you'd iterate based on monitoring and feedback.

Key Points to Mention

  • Durable state management using event sourcing or write-ahead logs with periodic snapshots.
  • Idempotency keys and exactly-once semantics for critical operations.
  • Supervision strategies: hierarchical supervisors, health checks, and automatic restarts.
  • Inter-agent communication via message queues (e.g., Kafka, RabbitMQ) or shared blackboard with versioning.
  • Observability: distributed tracing (OpenTelemetry), structured logging, and custom metrics for agent behavior.
  • Cost management: token usage tracking, budget caps, and rate limiting per agent.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.