← American Express Interview Insights

American Express·AI Engineer·Technical Phone Screen·Senior

Senior
May 2026

Summary

Technical interview at American Express for an AI Engineer role, focused almost entirely on how you evaluate LLM and agent systems end-to-end. Pretty deep dive, not the usual ML fundamentals stuff.

Questions Asked (6)

Q1

How would you design an end-to-end evaluation framework for an LLM or AI agent system, covering both offline and online metrics?

System DesignProduct Analytics & MetricsTechnical Trade-offs
Author's notes

This was the core question and it sprawled in a bunch of directions.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the system's goals and constraints, then propose a layered evaluation framework that combines offline benchmarks with online monitoring and A/B testing. Emphasize iterative improvement, human-in-the-loop validation, and alignment with business metrics like customer satisfaction and fraud detection.

Pro tip: Highlight the importance of defining clear, measurable objectives and establishing a baseline before deployment; also mention the need for continuous monitoring to detect drift and ensure compliance in a regulated industry like finance.

1. Define Objectives and Metrics

Identify the system's purpose, user needs, and business goals to select relevant offline and online metrics. Consider accuracy, latency, fairness, and business KPIs like conversion or fraud reduction.

2. Offline Evaluation

Use curated datasets and benchmarks to measure model performance, robustness, and bias. Employ techniques like cross-validation, adversarial testing, and human evaluation for subjective tasks.

3. Online Evaluation

Deploy the system in a controlled environment (e.g., A/B test, shadow mode) to collect real user interactions and measure online metrics. Monitor for drift, latency, and user feedback.

4. Iterate and Improve

Analyze results from both offline and online evaluations to identify gaps and retrain or fine-tune the model. Incorporate human feedback and update the evaluation framework as needed.

5. Governance and Compliance

Ensure the framework includes checks for regulatory compliance, explainability, and fairness, especially in finance. Establish a process for auditing and reporting.

Key Points to Mention

  • Offline metrics: accuracy, F1, BLEU, ROUGE, perplexity, bias metrics
  • Online metrics: user engagement, task success rate, latency, error rates, A/B test results
  • Human evaluation and feedback loops
  • Monitoring for data drift and model degradation
  • Business impact metrics (e.g., customer satisfaction, fraud detection rates)
  • Regulatory and ethical considerations (fairness, explainability, compliance)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

When using an LLM as a judge to score outputs, how do you make sure the judge itself is calibrated and trustworthy?

A/B Testing & ExperimentationTechnical Trade-offs
Author's notes

Blanked for a second on the calibration piece specifically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that LLM judges are fallible and need validation. Then describe a systematic process: define clear criteria, collect human-labeled ground truth, measure agreement, and continuously monitor. Emphasize that calibration is iterative and requires both quantitative metrics and qualitative error analysis.

Pro tip: Use a small, high-quality human-labeled dataset to compute agreement metrics like Cohen's kappa or Krippendorff's alpha, and set a threshold for acceptable agreement before trusting the judge. Also, regularly inject adversarial or edge cases to test robustness.

1. Define evaluation criteria and rubric

Clearly specify what aspects the judge should score (e.g., relevance, factuality, safety) and provide a detailed rubric with examples. This reduces ambiguity and aligns the judge with human expectations.

2. Collect human-labeled ground truth

Have multiple human annotators score a representative sample of outputs. Use majority voting or adjudication to resolve disagreements, creating a reliable reference dataset.

3. Measure agreement between judge and humans

Compute inter-annotator agreement (IAA) among humans to establish a ceiling, then calculate the judge's agreement with the human consensus using metrics like accuracy, F1, or correlation. Compare against the IAA ceiling.

4. Analyze errors and iterate

Examine cases where the judge disagrees with humans to identify patterns (e.g., bias, misunderstanding of criteria). Refine the prompt, rubric, or model, and re-evaluate.

5. Monitor and recalibrate in production

Continuously sample outputs and periodically re-validate the judge with fresh human labels. Set up alerts for drift in agreement metrics and update the judge as needed.

Key Points to Mention

  • Use of human-labeled ground truth and inter-annotator agreement to establish a baseline
  • Quantitative metrics such as Cohen's kappa, Krippendorff's alpha, or correlation coefficients
  • Qualitative error analysis to uncover systematic biases or misunderstandings
  • Adversarial testing and edge cases to probe robustness
  • Continuous monitoring and periodic recalibration to detect drift
  • Consideration of cost and scalability trade-offs when involving human annotators

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What metrics do you track specifically for agent systems, like tool use and plan execution?

Product Analytics & MetricsSystem Design
Author's notes

Covered tool call accuracy and plan-execution success rate, also touched on error recovery.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing metrics around the core agent loop: planning, tool selection, execution, and outcome. Then, for each stage, describe specific metrics you track and how they tie to business goals like reliability, efficiency, and user experience. Emphasize that you balance technical metrics with product-level KPIs to ensure the agent delivers value.

Pro tip: Highlight that you also monitor 'silent failures'—cases where the agent completes a task but with suboptimal tool use or plan deviations—by logging detailed traces and using LLM-based evaluators. This shows you go beyond surface-level metrics and care about root-cause analysis.

1. Define the agent lifecycle stages

Break down the agent's operation into distinct phases: planning, tool selection, tool execution, and final response generation. This provides a structured way to attach metrics.

2. Map metrics to each stage

For each stage, identify quantitative metrics. For planning: plan validity, step count, and goal completion rate. For tool use: tool selection accuracy, API success rate, and latency. For execution: task success rate, error rates, and retry counts.

3. Connect to business and user outcomes

Tie technical metrics to product KPIs such as user satisfaction (CSAT), task completion time, cost per task, and containment rate. Explain how these drive decisions.

4. Describe monitoring and iteration

Explain how you collect these metrics (e.g., logging, tracing, LLM-as-judge) and use them to iterate on prompts, tools, or model selection. Mention A/B testing and alerting on anomalies.

Key Points to Mention

  • Plan validity and goal completion rate to measure planning quality
  • Tool selection accuracy and API success rate to evaluate tool use
  • Task success rate and error rates to assess execution reliability
  • Latency and cost per task to track efficiency
  • User satisfaction (CSAT) and containment rate to link to business impact
  • Silent failure detection using LLM-based evaluators and trace analysis

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How do you handle regression testing when you swap out or upgrade the underlying model?

Technical Trade-offsA/B Testing & Experimentation
Author's notes

Short answer: version your eval sets, lock a baseline, diff against it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by emphasizing that regression testing for model swaps must go beyond accuracy metrics to include business-critical outcomes and fairness. Describe a structured process: define a golden dataset, run shadow deployments, compare against the old model with statistical tests, and gradually roll out with A/B testing. Highlight the importance of monitoring and having a rollback plan.

Pro tip: In regulated industries like finance, always include fairness and compliance checks in your regression suite—this shows you understand the stakes beyond pure performance. Also, automate the regression pipeline so it can be rerun quickly for every candidate model.

1. Define success criteria and baseline

Establish clear metrics (e.g., accuracy, F1, latency, fairness) and capture the current model's performance on a fixed golden dataset. Ensure these metrics align with business KPIs.

2. Build a comprehensive regression suite

Create a test suite covering functional correctness, edge cases, bias/fairness, and robustness. Include both automated unit tests and end-to-end scenario tests.

3. Run shadow deployment and compare

Deploy the new model in shadow mode alongside the old one, feeding it live traffic without affecting users. Compare outputs statistically to detect regressions.

4. Conduct A/B test with gradual rollout

If shadow results are satisfactory, run a controlled A/B test with a small user segment, monitoring key metrics and guardrails. Gradually increase traffic as confidence grows.

5. Monitor, iterate, and prepare rollback

Continuously monitor post-deployment for drift or unexpected issues. Have a rollback plan and automated alerts to revert quickly if regressions appear.

Key Points to Mention

  • Golden dataset and versioning of test data
  • Statistical significance testing (e.g., t-test, bootstrap) to compare models
  • Shadow deployment and canary releases
  • A/B testing framework with guardrail metrics
  • Fairness, bias, and compliance checks (critical for finance)
  • Automated CI/CD pipeline for regression testing
  • Rollback strategy and monitoring for model drift

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How do you balance automated evaluations with human evaluation, and when do you trust one over the other?

Technical Trade-offsProduct Analytics & Metrics
Author's notes

My take was that automated evals give you scale and speed but human evals are your ground truth calibration signal.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that automated and human evaluations serve complementary roles: automation for scale, speed, and regression detection, and humans for nuanced judgment, edge cases, and business context. Then describe a tiered evaluation strategy where automated metrics act as a first filter and human review is triggered for high-stakes or ambiguous cases, with clear criteria for when to trust each. Finally, tie your answer to American Express's context by emphasizing risk management, regulatory compliance, and customer trust.

Pro tip: Mention that you regularly calibrate automated metrics against human judgments (e.g., by measuring inter-annotator agreement or correlation with human ratings) to ensure your automated evaluations remain trustworthy over time. This shows you don't blindly trust either method and understand the importance of continuous validation.

1. Define the evaluation goals and constraints

Clarify what you're evaluating (e.g., model accuracy, fairness, safety) and the constraints (budget, time, regulatory requirements). This determines the appropriate mix of automated and human evaluation.

2. Leverage automated evaluations for scale and speed

Use automated metrics (e.g., precision/recall, BLEU, perplexity, fairness metrics) for continuous monitoring, regression testing, and large-scale comparisons. They are fast, cheap, and reproducible.

3. Use human evaluation for nuance and high-stakes decisions

Employ human reviewers for subjective quality, edge cases, ethical considerations, and final validation before deployment, especially in regulated industries like finance.

4. Establish triggers and thresholds for human review

Define clear criteria (e.g., low confidence scores, significant deviation from expected metrics, new data distributions) that automatically escalate to human evaluation.

5. Continuously calibrate and improve both methods

Regularly compare automated metrics with human judgments to detect drift, refine automated metrics, and update human guidelines. This ensures both remain aligned with business objectives.

Key Points to Mention

  • Complementary strengths: automation for scale/reproducibility, humans for context/nuance
  • Cost-benefit trade-offs: automated evaluations are cheaper and faster but may miss subtle issues
  • Risk-based approach: higher risk (e.g., financial decisions) warrants more human oversight
  • Calibration: measure correlation between automated metrics and human ratings to validate trust
  • Regulatory and compliance requirements in finance (e.g., fair lending, explainability) often mandate human review
  • Continuous monitoring and feedback loops to detect model drift and update evaluation strategies

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

What are the main pitfalls in LLM evaluation, like eval set contamination or distribution drift?

Root Cause AnalysisTechnical Trade-offsA/B Testing & Experimentation
Author's notes

Talked through contamination (train data leaking into your eval set), distribution drift over time, and how you can accidentally overfit your whole system to whatever your LLM judge rewards.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that LLM evaluation is inherently challenging due to the open-ended nature of tasks and the dynamic data environment. Then, systematically categorize the main pitfalls into data-related, model-related, and metric-related issues, providing concrete examples for each. Finally, discuss mitigation strategies and emphasize the importance of continuous monitoring and robust experimental design, especially in a regulated industry like finance.

Pro tip: Highlight that in high-stakes domains like finance, evaluation must go beyond accuracy to include fairness, robustness, and compliance metrics, and that A/B testing in production with guardrail metrics is essential to catch distribution shifts.

1. Define the evaluation goal and context

Clarify what you are evaluating (e.g., model performance, safety, fairness) and the specific use case, as pitfalls vary by application. This sets the stage for a targeted discussion.

2. Identify data-related pitfalls

Discuss contamination (train-test overlap), distribution drift (covariate shift, label shift), and data leakage. Explain how these can inflate performance and mislead deployment decisions.

3. Address model and metric pitfalls

Cover issues like overfitting to the eval set, metric misalignment (e.g., BLEU vs. human judgment), and the lack of standardized benchmarks for generative tasks. Mention the challenge of evaluating hallucinations and bias.

4. Propose mitigation and monitoring strategies

Suggest techniques such as dynamic eval sets, adversarial testing, human-in-the-loop evaluation, and continuous monitoring with statistical process control. Emphasize A/B testing with guardrail metrics to detect drift in production.

5. Tie back to business impact and compliance

Connect the pitfalls to real-world consequences like degraded customer experience, regulatory penalties, and reputational damage. Highlight the need for governance and audit trails in financial services.

Key Points to Mention

  • Eval set contamination: overlap between training and evaluation data, leading to inflated metrics; mitigation includes rigorous data splitting and deduplication.
  • Distribution drift: changes in data distribution over time (covariate shift, concept drift) that degrade model performance; requires monitoring and periodic re-evaluation.
  • Metric limitations: reliance on automated metrics (e.g., perplexity, BLEU) that may not correlate with human judgment or business KPIs; need for human evaluation and task-specific metrics.
  • Overfitting to the eval set: iterative model selection on the same eval set can lead to overfitting; use of holdout sets and cross-validation.
  • Lack of standardized benchmarks: especially for generative tasks, making comparisons difficult; need for domain-specific benchmarks and clear evaluation protocols.
  • Production monitoring and A/B testing: importance of guardrail metrics, canary deployments, and feedback loops to detect and mitigate issues in real-time.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.