← American Express Interview Insights
This was the core question and it sprawled in a bunch of directions.
Start by clarifying the system's goals and constraints, then propose a layered evaluation framework that combines offline benchmarks with online monitoring and A/B testing. Emphasize iterative improvement, human-in-the-loop validation, and alignment with business metrics like customer satisfaction and fraud detection.
Pro tip: Highlight the importance of defining clear, measurable objectives and establishing a baseline before deployment; also mention the need for continuous monitoring to detect drift and ensure compliance in a regulated industry like finance.
Identify the system's purpose, user needs, and business goals to select relevant offline and online metrics. Consider accuracy, latency, fairness, and business KPIs like conversion or fraud reduction.
Use curated datasets and benchmarks to measure model performance, robustness, and bias. Employ techniques like cross-validation, adversarial testing, and human evaluation for subjective tasks.
Deploy the system in a controlled environment (e.g., A/B test, shadow mode) to collect real user interactions and measure online metrics. Monitor for drift, latency, and user feedback.
Analyze results from both offline and online evaluations to identify gaps and retrain or fine-tune the model. Incorporate human feedback and update the evaluation framework as needed.
Ensure the framework includes checks for regulatory compliance, explainability, and fairness, especially in finance. Establish a process for auditing and reporting.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on the calibration piece specifically.
Start by acknowledging that LLM judges are fallible and need validation. Then describe a systematic process: define clear criteria, collect human-labeled ground truth, measure agreement, and continuously monitor. Emphasize that calibration is iterative and requires both quantitative metrics and qualitative error analysis.
Pro tip: Use a small, high-quality human-labeled dataset to compute agreement metrics like Cohen's kappa or Krippendorff's alpha, and set a threshold for acceptable agreement before trusting the judge. Also, regularly inject adversarial or edge cases to test robustness.
Clearly specify what aspects the judge should score (e.g., relevance, factuality, safety) and provide a detailed rubric with examples. This reduces ambiguity and aligns the judge with human expectations.
Have multiple human annotators score a representative sample of outputs. Use majority voting or adjudication to resolve disagreements, creating a reliable reference dataset.
Compute inter-annotator agreement (IAA) among humans to establish a ceiling, then calculate the judge's agreement with the human consensus using metrics like accuracy, F1, or correlation. Compare against the IAA ceiling.
Examine cases where the judge disagrees with humans to identify patterns (e.g., bias, misunderstanding of criteria). Refine the prompt, rubric, or model, and re-evaluate.
Continuously sample outputs and periodically re-validate the judge with fresh human labels. Set up alerts for drift in agreement metrics and update the judge as needed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered tool call accuracy and plan-execution success rate, also touched on error recovery.
Start by framing metrics around the core agent loop: planning, tool selection, execution, and outcome. Then, for each stage, describe specific metrics you track and how they tie to business goals like reliability, efficiency, and user experience. Emphasize that you balance technical metrics with product-level KPIs to ensure the agent delivers value.
Pro tip: Highlight that you also monitor 'silent failures'—cases where the agent completes a task but with suboptimal tool use or plan deviations—by logging detailed traces and using LLM-based evaluators. This shows you go beyond surface-level metrics and care about root-cause analysis.
Break down the agent's operation into distinct phases: planning, tool selection, tool execution, and final response generation. This provides a structured way to attach metrics.
For each stage, identify quantitative metrics. For planning: plan validity, step count, and goal completion rate. For tool use: tool selection accuracy, API success rate, and latency. For execution: task success rate, error rates, and retry counts.
Tie technical metrics to product KPIs such as user satisfaction (CSAT), task completion time, cost per task, and containment rate. Explain how these drive decisions.
Explain how you collect these metrics (e.g., logging, tracing, LLM-as-judge) and use them to iterate on prompts, tools, or model selection. Mention A/B testing and alerting on anomalies.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: version your eval sets, lock a baseline, diff against it.
Start by emphasizing that regression testing for model swaps must go beyond accuracy metrics to include business-critical outcomes and fairness. Describe a structured process: define a golden dataset, run shadow deployments, compare against the old model with statistical tests, and gradually roll out with A/B testing. Highlight the importance of monitoring and having a rollback plan.
Pro tip: In regulated industries like finance, always include fairness and compliance checks in your regression suite—this shows you understand the stakes beyond pure performance. Also, automate the regression pipeline so it can be rerun quickly for every candidate model.
Establish clear metrics (e.g., accuracy, F1, latency, fairness) and capture the current model's performance on a fixed golden dataset. Ensure these metrics align with business KPIs.
Create a test suite covering functional correctness, edge cases, bias/fairness, and robustness. Include both automated unit tests and end-to-end scenario tests.
Deploy the new model in shadow mode alongside the old one, feeding it live traffic without affecting users. Compare outputs statistically to detect regressions.
If shadow results are satisfactory, run a controlled A/B test with a small user segment, monitoring key metrics and guardrails. Gradually increase traffic as confidence grows.
Continuously monitor post-deployment for drift or unexpected issues. Have a rollback plan and automated alerts to revert quickly if regressions appear.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
My take was that automated evals give you scale and speed but human evals are your ground truth calibration signal.
Start by acknowledging that automated and human evaluations serve complementary roles: automation for scale, speed, and regression detection, and humans for nuanced judgment, edge cases, and business context. Then describe a tiered evaluation strategy where automated metrics act as a first filter and human review is triggered for high-stakes or ambiguous cases, with clear criteria for when to trust each. Finally, tie your answer to American Express's context by emphasizing risk management, regulatory compliance, and customer trust.
Pro tip: Mention that you regularly calibrate automated metrics against human judgments (e.g., by measuring inter-annotator agreement or correlation with human ratings) to ensure your automated evaluations remain trustworthy over time. This shows you don't blindly trust either method and understand the importance of continuous validation.
Clarify what you're evaluating (e.g., model accuracy, fairness, safety) and the constraints (budget, time, regulatory requirements). This determines the appropriate mix of automated and human evaluation.
Use automated metrics (e.g., precision/recall, BLEU, perplexity, fairness metrics) for continuous monitoring, regression testing, and large-scale comparisons. They are fast, cheap, and reproducible.
Employ human reviewers for subjective quality, edge cases, ethical considerations, and final validation before deployment, especially in regulated industries like finance.
Define clear criteria (e.g., low confidence scores, significant deviation from expected metrics, new data distributions) that automatically escalate to human evaluation.
Regularly compare automated metrics with human judgments to detect drift, refine automated metrics, and update human guidelines. This ensures both remain aligned with business objectives.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked through contamination (train data leaking into your eval set), distribution drift over time, and how you can accidentally overfit your whole system to whatever your LLM judge rewards.
Start by acknowledging that LLM evaluation is inherently challenging due to the open-ended nature of tasks and the dynamic data environment. Then, systematically categorize the main pitfalls into data-related, model-related, and metric-related issues, providing concrete examples for each. Finally, discuss mitigation strategies and emphasize the importance of continuous monitoring and robust experimental design, especially in a regulated industry like finance.
Pro tip: Highlight that in high-stakes domains like finance, evaluation must go beyond accuracy to include fairness, robustness, and compliance metrics, and that A/B testing in production with guardrail metrics is essential to catch distribution shifts.
Clarify what you are evaluating (e.g., model performance, safety, fairness) and the specific use case, as pitfalls vary by application. This sets the stage for a targeted discussion.
Discuss contamination (train-test overlap), distribution drift (covariate shift, label shift), and data leakage. Explain how these can inflate performance and mislead deployment decisions.
Cover issues like overfitting to the eval set, metric misalignment (e.g., BLEU vs. human judgment), and the lack of standardized benchmarks for generative tasks. Mention the challenge of evaluating hallucinations and bias.
Suggest techniques such as dynamic eval sets, adversarial testing, human-in-the-loop evaluation, and continuous monitoring with statistical process control. Emphasize A/B testing with guardrail metrics to detect drift in production.
Connect the pitfalls to real-world consequences like degraded customer experience, regulatory penalties, and reputational damage. Highlight the need for governance and audit trails in financial services.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.