← American Express Interview Insights

American Express·AI Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Amex for an AI Engineer role, focused almost entirely on memory architecture for AI agents. It went deeper than I expected and I definitely undersold a few areas.

Questions Asked (7)

Q1

How would you design the memory system for an AI agent, covering both short-term working memory and durable long-term storage?

System DesignTechnical Trade-offs
Author's notes

I started with context windows as working memory and then talked about external stores for persistence.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then propose a layered memory architecture that separates short-term working memory from long-term storage. Discuss trade-offs around latency, cost, consistency, and retrieval accuracy, and tie your design to American Express's needs like compliance, security, and personalization.

Pro tip: Emphasize that memory is not just storage—it's about retrieval relevance and forgetting. Show you understand that over-retaining context can degrade performance and increase cost, so design for selective retention and summarization.

1. Clarify Requirements and Constraints

Ask about use cases, latency budgets, data sensitivity, and scale. Identify what the agent needs to remember and for how long.

2. Design Short-Term Working Memory

Propose an in-memory store (e.g., Redis) with a sliding window or summarization to keep recent context. Discuss token limits and eviction policies.

3. Design Long-Term Durable Storage

Suggest a vector database for semantic retrieval plus a relational or document store for structured facts. Include embedding, indexing, and retrieval strategies.

4. Define Memory Management Policies

Explain how memories are written, consolidated, summarized, and forgotten. Cover TTL, relevance scoring, and user control.

5. Address Trade-offs and Non-Functional Requirements

Discuss latency vs. accuracy, cost vs. scalability, and compliance (e.g., PII handling, audit trails). Tie back to American Express's regulatory environment.

Key Points to Mention

  • Vector databases (e.g., Pinecone, Weaviate) for semantic long-term memory
  • Short-term memory with sliding window and summarization to manage context length
  • Retrieval-augmented generation (RAG) for grounding responses in stored knowledge
  • Memory consolidation and forgetting policies to avoid bloat and maintain relevance
  • Security and compliance: encryption, access controls, and PII redaction
  • Trade-offs: latency vs. recall, cost of storage vs. performance, consistency models

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

What's the distinction between episodic and semantic memory in an agent context, and how does that affect your storage design?

System DesignData Modeling
Author's notes

Knew this one cold.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly defining episodic and semantic memory in the context of AI agents, then explain how their differences (e.g., temporal specificity vs. general knowledge) drive distinct storage requirements. Finally, connect this to a concrete storage design that balances performance, scalability, and compliance, especially in a regulated industry like finance.

Pro tip: Emphasize that episodic memory often requires immutable, time-series storage for auditability, while semantic memory benefits from vector databases for efficient similarity search. This shows you understand both technical and regulatory nuances.

1. Define episodic and semantic memory

Explain that episodic memory stores specific events or interactions tied to time and context, while semantic memory stores general facts and concepts independent of when they were learned.

2. Highlight key differences

Contrast them on dimensions like temporal relevance, retrieval patterns, and update frequency: episodic is append-only and time-ordered; semantic is mutable and accessed by similarity or association.

3. Map to storage requirements

Derive storage needs: episodic memory needs efficient time-range queries, immutability, and possibly TTL; semantic memory needs fast approximate nearest neighbor search and support for updates.

4. Propose a storage design

Suggest a hybrid architecture: e.g., a time-series database or event store for episodic memory, and a vector database or knowledge graph for semantic memory, with a synchronization mechanism.

5. Address trade-offs and compliance

Discuss trade-offs like cost, latency, and consistency, and mention compliance considerations (e.g., data retention policies, PII handling) relevant to American Express.

Key Points to Mention

  • Episodic memory is event-based and time-stamped; semantic memory is fact-based and context-independent.
  • Storage for episodic memory: time-series DB, event sourcing, or append-only logs with indexing on timestamps.
  • Storage for semantic memory: vector databases (e.g., Pinecone, Weaviate) for embeddings, or graph databases for relationships.
  • Retrieval differences: episodic uses temporal queries; semantic uses similarity search or graph traversal.
  • Scalability and performance: episodic grows unboundedly, so consider tiered storage or summarization; semantic requires efficient indexing for high-dimensional vectors.
  • Compliance: ensure audit trails for episodic memory and data privacy for semantic memory, especially in finance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Walk me through the retrieval strategies you'd consider for an agent's long-term memory. When would you use a vector store versus a structured database versus hierarchical summaries?

System DesignTechnical Trade-offs
Author's notes

Vector store for fuzzy semantic search, structured DB when you need exact lookups or filtering on metadata, hierarchical summaries to compress episodic history without losing the gist.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing long-term memory as a multi-tier system where each retrieval strategy serves a distinct purpose: vector stores for semantic recall of unstructured experiences, structured databases for precise, transactional facts, and hierarchical summaries for compressed, high-level context. Then walk through a decision framework based on query type, latency, and consistency requirements, and tie it back to American Express use cases like customer interaction history and compliance.

Pro tip: Emphasize that in production you rarely use one strategy in isolation—you combine them with a router or fallback logic, and you must address memory staleness, eviction, and privacy (e.g., PII redaction) upfront, which shows you think beyond the happy path.

1. Clarify memory types and access patterns

Distinguish between episodic (past interactions), semantic (facts/knowledge), and procedural (skills) memory, and identify whether queries are similarity-based, exact-match, or need aggregation.

2. Map retrieval strategies to memory types

Explain that vector stores excel for semantic similarity over unstructured text, structured DBs for precise, relational, or time-series facts, and hierarchical summaries for compressing long histories into digestible context.

3. Define selection criteria and trade-offs

Discuss factors like query latency, accuracy, cost, scalability, and consistency; for example, vector search is approximate and fast but may miss exact matches, while SQL is exact but slower for fuzzy queries.

4. Propose a hybrid architecture with routing

Describe how an agent could route queries: use a classifier or rules to decide which store to hit, or combine results (e.g., vector for context, DB for facts) and merge them.

5. Address operational concerns and Amex context

Cover memory lifecycle (TTL, summarization triggers), privacy (PII handling), and compliance (audit trails), and relate to American Express scenarios like fraud detection or personalized recommendations.

Key Points to Mention

  • Vector stores: ideal for semantic search over unstructured data (e.g., chat logs, documents) using embeddings; trade-offs include approximate recall and embedding drift.
  • Structured databases: best for exact, transactional, or relational data (e.g., user profiles, transaction history) with ACID guarantees; can be slow for fuzzy queries.
  • Hierarchical summaries: useful for compressing long-term memory into multi-level abstractions (e.g., daily → weekly → monthly summaries) to fit context windows and reduce retrieval cost.
  • Hybrid retrieval: combine strategies with a router or fallback; e.g., use vector search to find relevant memories, then fetch exact details from a DB.
  • Memory lifecycle: implement TTL, eviction policies, and periodic summarization to manage growth and staleness.
  • Evaluation metrics: measure retrieval precision/recall, latency, and impact on downstream agent performance; consider A/B testing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you handle forgetting or compaction in an agent's memory? What policies would you apply and why?

System DesignTechnical Trade-offs
Author's notes

Recency bias, relevance scoring, explicit TTLs on episodic records.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing memory management as a trade-off between recall accuracy, latency, and cost, then propose a tiered policy combining forgetting (TTL, importance-based eviction) and compaction (summarization, clustering). Emphasize that policies should be driven by business requirements like compliance and personalization, and include monitoring and feedback loops.

Pro tip: Tie your answer to American Express's context: mention that financial regulations may require retaining certain interactions for years, so forgetting policies must be auditable and configurable, not just heuristic.

1. Clarify requirements and constraints

Ask about use case, data sensitivity, regulatory retention needs, and performance SLAs to ground your policy choices.

2. Define forgetting policies

Propose time-based (TTL), importance-based (recency, frequency, user feedback), and capacity-based eviction, with configurable thresholds.

3. Define compaction policies

Suggest summarization, clustering similar memories, and abstraction to reduce storage while preserving key information.

4. Implement tiered memory architecture

Design hot/warm/cold storage tiers with different retention and compaction strategies to balance cost and latency.

5. Monitor, evaluate, and iterate

Set up metrics (e.g., recall@k, latency, cost) and A/B tests to refine policies based on real-world performance.

Key Points to Mention

  • Trade-offs between memory size, retrieval accuracy, latency, and cost
  • Regulatory compliance (e.g., GDPR, financial record-keeping) and auditability
  • Importance scoring using recency, frequency, and user feedback
  • Summarization and abstraction techniques for compaction
  • Tiered storage (hot/warm/cold) and caching strategies
  • Monitoring and feedback loops to adapt policies over time

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you design the read and write APIs for an agent's memory layer?

API & IntegrationsSystem Design
Author's notes

Talked through a write API that handles both episodic writes and semantic upserts separately, and a read API with query params for recency, relevance score, and memory type.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements for the agent's memory layer, such as what types of memory (episodic, semantic, procedural) and access patterns are needed. Then propose a RESTful API design with clear endpoints for reading and writing memory, emphasizing scalability, consistency, and security. Finally, discuss trade-offs and how you would handle challenges like concurrency, versioning, and data privacy.

Pro tip: Demonstrate awareness of American Express's regulatory environment by mentioning data encryption, access controls, and audit logging in your API design. Also, highlight how you would optimize for low-latency reads and high-throughput writes, which are critical for real-time agent interactions.

1. Clarify Requirements

Ask questions to understand the agent's use cases, memory types, expected read/write patterns, and non-functional requirements like latency, throughput, and compliance.

2. Define API Resources and Operations

Outline the core resources (e.g., memories, sessions) and map CRUD operations to HTTP methods, ensuring intuitive and consistent endpoint naming.

3. Design for Scalability and Performance

Propose techniques like caching, pagination, and asynchronous writes to handle high loads, and discuss sharding or partitioning strategies for memory storage.

4. Address Security and Compliance

Incorporate authentication, authorization, encryption, and audit trails to meet financial industry standards and protect sensitive data.

5. Discuss Trade-offs and Evolution

Acknowledge trade-offs between consistency and availability, and explain how the API can evolve with versioning and backward compatibility.

Key Points to Mention

  • RESTful design principles with clear resource modeling and HTTP semantics
  • Support for different memory types (episodic, semantic, procedural) and their access patterns
  • Scalability considerations: caching, pagination, rate limiting, and asynchronous processing
  • Security measures: OAuth 2.0, encryption at rest and in transit, role-based access control
  • Compliance with financial regulations: audit logging, data retention policies, and privacy
  • Trade-offs: consistency vs. availability, latency vs. durability, and versioning strategies

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How do you maintain consistency in shared memory when multiple agents or concurrent sessions are reading and writing at the same time?

System DesignTechnical Trade-offs
Author's notes

This was the hardest part of the whole interview.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the context: shared memory among multiple agents or sessions often involves concurrent reads/writes to a common data store. Then discuss a layered strategy: define consistency requirements, choose appropriate synchronization primitives (locks, transactions, optimistic concurrency), and handle conflicts with versioning or conflict-free replicated data types (CRDTs). Emphasize trade-offs between consistency, latency, and availability, and tie back to American Express's need for reliability and data integrity.

Pro tip: Mention that in practice, you'd avoid shared mutable state where possible and prefer message passing or event sourcing; when shared state is necessary, use optimistic concurrency with version checks and idempotent operations to minimize locking overhead.

1. Clarify consistency requirements

Determine whether strong consistency (linearizability) or eventual consistency is acceptable for the use case, as this drives the entire design.

2. Choose a concurrency control mechanism

Select appropriate techniques such as pessimistic locking, optimistic concurrency control (versioning/ETags), or transactional memory, balancing performance and correctness.

3. Handle conflicts and failures

Define conflict resolution strategies (e.g., last-write-wins, merge functions, CRDTs) and ensure fault tolerance with retries, idempotency, and deadlock avoidance.

4. Implement and monitor

Use atomic operations, memory barriers, or database transactions as needed, and add monitoring for contention, latency, and consistency violations.

5. Evaluate trade-offs

Discuss how the chosen approach impacts scalability, latency, and complexity, and justify decisions based on business requirements.

Key Points to Mention

  • Optimistic vs. pessimistic concurrency control and when to use each
  • Versioning, timestamps, or ETags for detecting write conflicts
  • Atomic operations and memory barriers in low-level shared memory
  • Distributed transactions or two-phase commit for multi-agent systems
  • CRDTs and eventual consistency for high availability
  • Idempotent operations and retry logic to handle transient conflicts

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q7

How would you evaluate whether your agent's memory system is actually working well in production?

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

Retrieval precision and recall against a labeled set of queries, latency percentiles on read paths, and downstream task performance as the real signal.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining what 'working well' means for the agent's memory in terms of business and user outcomes, then outline a layered evaluation framework combining offline metrics, online A/B tests, and production monitoring. Emphasize how you would tie memory performance to downstream KPIs like task success, personalization quality, and customer satisfaction.

Pro tip: In production, memory systems often fail silently—so instrument leading indicators like retrieval hit rate and staleness, and set up guardrail metrics to catch regressions before they impact the customer experience.

1. Define success criteria and metrics

Translate 'working well' into measurable objectives: memory recall/precision, relevance, freshness, and impact on user outcomes (e.g., resolution rate, CSAT). Align these with American Express business goals like customer loyalty and operational efficiency.

2. Establish offline evaluation and baselines

Use curated datasets and simulated conversations to measure memory retrieval accuracy, context retention, and hallucination rates. Compare against a baseline (e.g., no memory or rule-based memory) to quantify improvement.

3. Run online A/B tests with guardrails

Deploy the memory system to a subset of traffic and measure impact on primary KPIs (e.g., task completion, customer effort score) while monitoring guardrail metrics (latency, cost, safety). Ensure statistical power and avoid novelty effects.

4. Monitor production health and drift

Track live metrics such as memory hit rate, staleness, and retrieval latency. Set up alerts for anomalies and periodically re-evaluate with shadow deployments or canary releases to detect model or data drift.

5. Iterate with feedback loops

Collect user feedback (explicit and implicit) and error analysis to identify failure modes. Use these insights to refine memory policies, update embeddings, or adjust retrieval thresholds, closing the loop between evaluation and improvement.

Key Points to Mention

  • Define clear offline and online metrics for memory performance (e.g., recall@k, MRR, task success rate).
  • Use A/B testing with proper randomization and guardrail metrics to measure causal impact.
  • Monitor production for data drift, staleness, and silent failures with leading indicators.
  • Tie memory evaluation to business KPIs like customer satisfaction, resolution time, and cost per interaction.
  • Incorporate human evaluation and error analysis to capture nuanced memory failures.
  • Consider privacy and compliance constraints when logging and evaluating memory data.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.