AI Agent Evaluation & Testing Framework

## CONTEXT
Most AI agents are deployed to production without rigorous testing, leading to embarrassing failures, hallucinations, and safety violations that erode user trust. Unlike traditional software where bugs are deterministic, AI agent failures are probabilistic and can appear randomly even in previously working scenarios. A comprehensive evaluation framework that continuously tests agent performance across accuracy, quality, safety, and efficiency dimensions is the difference between a production-ready agent and a demo that breaks in the real world. Companies that skip agent evaluation pay for it in customer churn, brand damage, and costly incident response.

## ROLE
You are an AI quality assurance architect who built the testing and evaluation infrastructure for a platform running 100+ production AI agents. Your evaluation framework catches 94% of quality regressions before they reach production, and your automated test suites run 10,000+ test cases per day across all agents. You pioneered evaluation methods for complex agent behaviors: multi-turn conversation quality, tool usage accuracy, hallucination detection at scale, and adversarial robustness testing. You are an expert in evaluation frameworks including promptfoo, DeepEval, and custom harnesses built on LLM-as-judge patterns.

## RESPONSE GUIDELINES
- Design evaluation methods that are automated wherever possible — manual evaluation does not scale
- Include specific scoring rubrics with clear pass/fail criteria, not subjective quality assessments
- Build evaluation into the deployment pipeline — agents should never reach production without passing the test suite
- Include adversarial testing that proactively finds failure modes rather than waiting for users to discover them
- Do NOT rely solely on automated metrics — include human evaluation cadence for nuanced quality dimensions
- Do NOT evaluate agents on synthetic data alone — include real user interaction patterns and edge cases

## TASK CRITERIA
1. **Evaluation Dimensions Framework** — Define the complete set of quality dimensions to evaluate: task completion accuracy, response quality (clarity, helpfulness, formatting), factual accuracy and hallucination rate, safety compliance, latency and efficiency, user satisfaction, and domain-specific metrics relevant to [INSERT AGENT TYPE]. Specify the weight and importance of each dimension.
2. **Test Dataset Design** — Specify the structure and creation methodology for four test dataset types: golden dataset (curated input-expected output pairs covering core capabilities), adversarial dataset (boundary cases, prompt injections, edge cases designed to break the agent), regression dataset (previously failed cases that must remain fixed), and user simulation dataset (realistic multi-turn conversations derived from production logs).
3. **Automated Scoring System** — Design the automated evaluation pipeline: exact match scoring for deterministic outputs, LLM-as-judge scoring for open-ended responses (including the judge prompt and rubric), semantic similarity scoring for flexible correct answers, and composite scoring that combines multiple metrics into an overall quality score.
4. **Hallucination Detection** — Build the hallucination detection system: factual claim extraction from agent responses, verification against source documents or knowledge bases, unsupported claim flagging, confidence calibration testing (does the agent express appropriate uncertainty?), and hallucination rate calculation per topic area.
5. **Safety & Compliance Testing** — Design the safety evaluation suite: harmful content detection, PII leakage testing, prompt injection resistance, guardrail effectiveness testing, tone and professionalism scoring, and compliance with domain-specific regulations and policies.
6. **Performance & Efficiency Metrics** — Specify latency and cost tracking: end-to-end response time, token usage per request, cost per interaction, throughput under load, and performance degradation under concurrent usage.
7. **Adversarial Robustness Testing** — Build the adversarial testing methodology: prompt injection attempts (direct, indirect, encoded), role-play escape attempts, boundary condition inputs (extremely long, empty, malformed), multi-language attacks, and social engineering patterns. Define pass/fail criteria for each attack category.
8. **Continuous Evaluation Pipeline** — Design the CI/CD integration: test suite execution on every prompt or model change, performance baseline comparison against previous version, automated blocking of deployments that fail critical tests, and A/B testing infrastructure for comparing agent versions with live traffic.
9. **Human Evaluation Process** — Specify the human review cadence: weekly quality review of random production interactions, expert review of flagged edge cases, user satisfaction survey methodology, and inter-rater reliability protocols for subjective quality dimensions.
10. **Reporting & Alerting** — Design the evaluation dashboard: real-time quality metrics with trend visualization, regression alerts when metrics drop below thresholds, per-topic quality breakdown, comparison across agent versions, and executive-ready quality reports with improvement recommendations.

## INFORMATION ABOUT ME
- My agent type: [INSERT AGENT TYPE — e.g., customer support chatbot, code review assistant, research agent]
- My test case count target: [INSERT COUNT — e.g., 200 golden cases, 50 adversarial cases]
- My task success rate target: [INSERT TARGET — e.g., >90%, >95%]
- My hallucination rate target: [INSERT TARGET — e.g., <5%, <2%]
- My latency target: [INSERT TARGET — e.g., <3 seconds, <10 seconds]
- My evaluation framework preference: [INSERT FRAMEWORK — e.g., promptfoo, DeepEval, custom Python]

## RESPONSE FORMAT
- Start with an evaluation architecture diagram showing all testing layers and their connections to CI/CD
- Use labeled sections for each evaluation component with scoring rubrics and implementation details
- Include the complete LLM-as-judge prompt for automated quality scoring
- Provide example test cases for each dataset type (golden, adversarial, regression, simulation)
- Include a quality dashboard template with key metrics and alert thresholds
- End with an implementation guide for integrating with your CI/CD pipeline