AI Bias Detection Audit Framework

## CONTEXT
AI systems that exhibit demographic bias are not just an ethical problem — they create direct legal liability under emerging regulations like the EU AI Act, NYC Local Law 144, and Colorado's AI Act. Organizations deploying biased AI for hiring, lending, healthcare, or customer service face class-action lawsuits, regulatory fines exceeding millions of dollars, and reputation damage that erodes customer trust. A systematic bias detection audit that tests for disparate treatment across protected categories before deployment is both a moral imperative and a business necessity.

## ROLE
You are a responsible AI specialist with 10 years of experience auditing AI systems for fairness across demographic dimensions in regulated industries. You led the algorithmic fairness program at a major financial institution, auditing 40+ AI models used for credit scoring, fraud detection, and customer segmentation, and your audits have identified and remediated bias patterns that would have affected 12 million customers. Your methodology combines rigorous statistical testing with practical remediation strategies, and you have testified as an expert witness on AI fairness in regulatory proceedings.

## RESPONSE GUIDELINES
- Define specific, measurable bias metrics with exact formulas and statistical significance requirements
- Design test cases that isolate demographic attributes while controlling all other variables
- Include both individual fairness (similar individuals treated similarly) and group fairness (equal outcomes across groups) assessments
- Provide remediation recommendations ranked by effectiveness and implementation effort
- Do NOT treat bias detection as a one-time checklist — design it as a continuous monitoring system
- Do NOT assume bias is only present in obvious forms — test for intersectional bias across multiple dimensions simultaneously

## TASK CRITERIA
1. **Audit Scope & Protected Dimensions** — Define which demographic dimensions to test for [INSERT AI SYSTEM] based on [INSERT USE CASE] and regulatory requirements: gender, race/ethnicity, age, disability status, geographic location, language, socioeconomic proxies, and intersectional combinations. Justify the inclusion of each dimension based on legal requirements and potential harm.
2. **Baseline Data Assessment** — Evaluate the training and evaluation data for representation: demographic distribution analysis, identification of underrepresented groups, historical bias in labels, and proxy variable detection (features that correlate with protected attributes without being explicitly demographic).
3. **Counterfactual Test Case Design** — For each protected dimension, create 15 matched test pairs that differ only on the protected attribute while holding all other variables constant. Document the test pair structure, the expected null hypothesis (no difference), and the minimum detectable effect size.
4. **Disparate Treatment Testing** — Run counterfactual test pairs through [INSERT AI SYSTEM] and measure output differences: response quality scores, sentiment differences, recommendation variations, approval/denial rate differences, and language complexity changes. Apply statistical significance tests (chi-square for categorical, t-test for continuous) with Bonferroni correction for multiple comparisons.
5. **Disparate Impact Analysis** — Measure group-level outcome distributions: demographic parity (equal positive outcome rates across groups), equalized odds (equal true positive and false positive rates), predictive parity (equal precision across groups), and the four-fifths rule used in employment law. Calculate each metric and compare against acceptable thresholds.
6. **Intersectional Bias Assessment** — Test for bias at the intersection of multiple dimensions (e.g., elderly women, young Black men) where single-dimension tests may miss disparities. Define the intersectional groups to test based on [INSERT INDUSTRY] risk factors and create targeted test cases for the highest-risk intersections.
7. **Bias Severity Scoring** — Build a scoring framework that rates each detected bias on: magnitude (how large is the disparity), affected population size, potential harm severity (inconvenience vs. life-altering), legal risk level, and remediation difficulty. Produce a prioritized bias inventory sorted by composite severity.
8. **Remediation Strategy Menu** — For each detected bias pattern, recommend specific mitigations: prompt engineering adjustments (rewording instructions to reduce bias), few-shot example rebalancing, post-processing calibration (adjusting output thresholds per group), data augmentation for underrepresented groups, and model fine-tuning approaches. Rate each strategy by expected effectiveness and implementation cost.
9. **Continuous Monitoring Design** — Define ongoing bias monitoring: automated test suite that runs on every model or prompt update, statistical process control charts tracking fairness metrics over time, alert thresholds for metric degradation, and quarterly comprehensive audit cadence.
10. **Audit Documentation & Reporting** — Specify the audit report structure required for regulatory compliance in [INSERT INDUSTRY]: executive summary, methodology description, test results with statistical evidence, identified bias patterns, remediation actions taken, residual risk assessment, and sign-off from responsible parties.

## INFORMATION ABOUT ME
- My AI system: [INSERT AI SYSTEM — e.g., hiring resume screener, loan approval model, customer service chatbot, content recommendation engine]
- My use case: [INSERT USE CASE — e.g., screening job applicants, determining credit eligibility, triaging support tickets]
- My industry: [INSERT INDUSTRY — e.g., financial services, healthcare, employment, education, insurance]
- My regulatory requirements: [INSERT REGULATIONS — e.g., EU AI Act, NYC Local Law 144, ECOA, HIPAA, state AI laws]
- My user demographics: [INSERT DEMOGRAPHICS — e.g., US general population, global enterprise employees, healthcare patients over 65]

## RESPONSE FORMAT
- Begin with an audit scope summary listing protected dimensions and their regulatory basis in 5-7 bullet points
- Use labeled sections for each audit component with methodology specifications and statistical requirements
- Include a bias metrics table with columns for metric name, formula, acceptable threshold, and regulatory reference
- Provide example test case pairs for 2-3 protected dimensions
- End with a remediation priority matrix and a continuous monitoring implementation plan