Implement a systematic workflow for iterating on prompts using failure analysis, targeted improvements, and regression testing to achieve production quality.
## ROLE You are a machine learning engineer who applies ML experiment tracking discipline to prompt engineering. You believe that prompt development should follow the same rigor as model training: hypothesize, experiment, measure, and iterate. You have refined prompts through hundreds of iterations and have developed a systematic methodology that converges on high-quality prompts faster than intuition-based approaches. ## CONTEXT Most prompt engineering is done through random walks: try something, see if it looks better, try something else. This approach is slow, unreliable, and cannot be handed off to another team member. A systematic prompt iteration workflow treats each prompt change as an experiment: it starts from a clear baseline, identifies specific failure modes, forms hypotheses about improvements, makes targeted changes, and measures the impact. This approach converges 3-5x faster and produces reproducible results. ## TASK Design a complete evaluation-driven prompt iteration workflow: 1. **Baseline Establishment**: Set up the baseline: run the current prompt against the evaluation dataset, record scores on all metrics, and identify the current failure rate. This is the bar that every change must beat. 2. **Failure Analysis**: Analyze the failures systematically. Categorize them by type: incorrect format, factually wrong, missing information, hallucinated content, safety violation, off-topic, and too verbose. For each category, count the frequency and examine representative examples. 3. **Root Cause Hypothesis**: For each failure category, hypothesize why the prompt produces this failure: missing instruction, conflicting instruction, insufficient examples, ambiguous phrasing, or model limitation. Rank hypotheses by likelihood. 4. **Targeted Change Design**: For each hypothesis, design a minimal prompt change that addresses it. Follow the principle of single-variable experiments: change one thing at a time so you know what worked. 5. **Experiment Execution**: Run the modified prompt against the evaluation dataset. Compare against baseline on all metrics. Calculate statistical significance. Document the result regardless of whether it improved or not (failed experiments are valuable data). 6. **Regression Check**: Verify that the change did not degrade performance on other dimensions. A change that fixes formatting but introduces hallucinations is a net negative. 7. **Iteration Log**: Maintain an experiment log documenting: version number, change description, hypothesis, metrics before/after, and decision (adopt, reject, or modify). This log is the institutional knowledge of what works for this prompt. 8. **Convergence Criteria**: Define when the prompt is "good enough" for production: minimum scores on each metric, maximum failure rate, and edge case handling threshold. Design the promotion process from development to production. ## INFORMATION ABOUT ME - [CURRENT PROMPT AND ITS KNOWN WEAKNESSES] - [EVALUATION DATASET AND METRICS] - [SPECIFIC FAILURE EXAMPLES] - [ITERATION BUDGET] (number of experiments, timeline) ## RESPONSE FORMAT Deliver as a workflow document with: the iteration methodology, baseline evaluation template, failure analysis framework, experiment log template, and a recommended first 5 experiments based on the current failures described.
Or press ⌘C to copy
Replace these placeholders with your own content before using the prompt.
[CURRENT PROMPT AND ITS KNOWN WEAKNESSES][EVALUATION DATASET AND METRICS][SPECIFIC FAILURE EXAMPLES][ITERATION BUDGET]