Table of Contents
Quick Answer
AI bias is systematic error that produces unfair outcomes for specific groups. Detection requires statistical fairness metrics (disparate impact, equalised odds, demographic parity), and mitigation spans pre-processing, in-processing, and post-processing techniques.
- Bias enters through data, model, and deployment stages
- No single fairness metric fits all contexts — choose based on harm
- Regulators (EEOC, ICO, CNIL, DPDP Board) now audit for algorithmic discrimination
What Is AI Bias?
AI bias occurs when an AI system produces outputs that systematically favour or disadvantage certain groups. The NIST Special Publication 1270 ("Towards a Standard for Identifying and Managing Bias in Artificial Intelligence", March 2022) categorises AI bias into three types:
- Systemic bias (historical, societal, institutional)
- Statistical bias (sampling, measurement, algorithmic)
- Human cognitive bias (confirmation, automation complacency)
Famous incidents include Amazon's scrapped hiring tool (2018), ProPublica's COMPAS investigation (2016), and Apple Card credit-limit disparities (2019).
Key Details / Requirements
Common Fairness Metrics
| Metric | Formula | When to Use |
|---|---|---|
| Demographic Parity | P(pred=1 or A=0) = P(pred=1 or A=1) | When base rates should be equal |
| Disparate Impact | P(pred=1 or A=0) / P(pred=1 or A=1) >= 0.8 | EEOC "four-fifths rule" |
| Equalised Odds | Equal TPR and FPR across groups | When label accuracy matters |
| Equal Opportunity | Equal TPR across groups | When false negatives harm |
| Calibration | Predicted probability = actual outcome | Risk scoring (recidivism, credit) |
Open-Source Detection Tools
| Tool | Maintainer | Best For |
|---|---|---|
| AIF360 | IBM / LF AI | 70+ fairness metrics, end-to-end pipeline |
| Fairlearn | Microsoft | Tabular data, disparity dashboards |
| What-If Tool | Google PAIR | Visual counterfactual analysis |
| Aequitas | University of Chicago | Bias audits for public policy |
| Facets | Visual feature-distribution analysis | |
| Themis-ML | Cornell | Integration with scikit-learn |
Real-World Examples / Case Studies
Amazon (2018) — Internal resume-screening AI down-weighted resumes containing "women's" (e.g., "women's chess club"); tool was scrapped.
Apple Card (2019) — New York DFS investigated alleged gender-based credit-limit disparities; Goldman Sachs (issuer) responded with process changes.
Dutch SyRI (2020) — The Hague District Court struck down the System Risk Indication welfare-fraud AI for violating ECHR Article 8.
UK A-level Algorithm (2020) — Ofqual's grading algorithm downgraded disadvantaged students; withdrawn after public outcry.
What This Means for AI Teams
Every production AI system in 2026 needs a documented fairness assessment. Regulators from the FTC to the European Data Protection Board explicitly cite bias audits as evidence of compliance. The EEOC's 2023 technical assistance and OFCCP's 2024 AI hiring guidance treat disparate impact analysis as non-negotiable.
Compliance Checklist
- Document protected characteristics relevant to your use case
- Run a pre-training data audit for representation and historical bias
- Choose fairness metrics matched to the harm profile
- Test across protected groups at each training checkpoint
- Build a dashboard for live monitoring of fairness drift
- Establish a human-review escalation path for contested decisions
- Publish a Model Card (Mitchell et al., 2019) documenting fairness evaluations
Conclusion
Bias audits are the new regulatory floor. Teams that embed fairness testing into CI/CD pipelines ship AI that courts, regulators, and customers trust.
Start your fairness audit with Misar AI's Bias Audit Kit — AIF360 and Fairlearn preloaded.