Bias & Data Quality
Bad data in, bad AI out — with the uncomfortable punchline that there is no "perfectly fair".
Note
What This Is About
A skin cancer algorithm detects changes on light skin with high accuracy — on dark skin, detection rates drop drastically. Is the algorithm racist? No. It simply hasn't seen enough training images of dark skin tones. The error is in the data, and the fix requires engineering, not ideology. This article shows why bias is a technical data quality problem bounded by mathematical constraints.
Training Bias
Definition
Training Bias
Analogy
Definition
Example: Dermatology Image Classifier
A neural network for skin cancer detection is trained on a dataset consisting predominantly of images of light skin (Fitzpatrick types I-III). The model's loss function converges optimally for the dominant pixel distribution. When evaluated on dark skin (Fitzpatrick types IV-VI), detection accuracy drops drastically. The reason is mathematical: the model has seen too few examples of dark skin conditions, so its learned feature representations are poorly calibrated for those inputs. The fix is targeted data engineering: collect and annotate images across all skin types until the training distribution matches the deployment distribution.
Caution
Misconception: "AI Is More Objective Than Humans"
This claim reverses the causality. An AI model does not generate its own judgments from first principles — it compresses human-generated data into statistical patterns. If the underlying data contains biases, the AI reproduces them with perfect consistency at massive scale. A biased loan officer might reject 100 applications per month based on flawed criteria. An AI trained on that officer's historical decisions replicates the same flawed criteria across millions of decisions. The AI doesn't add objectivity — it adds speed and consistency to whatever patterns exist in the data, correct or not.
Impossibility Theorem
Definition
Impossibility Theorem
Analogy
Definition
The proportion of incorrectly flagged cases (the "false alarms" from the building analogy) is identical for all groups. Distributes unfounded alarms equally — but may miss more actual cases in one group.
The proportion of missed positive cases (the "missed fires" from the building analogy) is identical for all groups. Distributes detection equally — but may lead to more false alarms in some groups.
Example: Medical Screening
A screening algorithm detects a condition that affects 8% of Population A and 4% of Population B. If the developer keeps the model calibrated and enforces equal false positive rates (both groups get the same rate of unnecessary follow-up tests), the mathematics forces unequal false negative rates: a larger share of actual cases will be missed in Population A. If the developer instead enforces equal false negative rates (no group has more missed cases), Population A will receive significantly more false alarms. The developer must make an explicit engineering decision — there is no setting that avoids both.
Caution
Misconception: "Bias Can Be Eliminated with Enough Effort"
The Impossibility Theorem refutes this directly. Bias reduction is always possible, but complete elimination across all fairness definitions simultaneously is not. This is not a limitation of current technology or compute power — it is a proven mathematical constraint. Any claim that a system has "eliminated bias" is either using a single narrow fairness metric (while violating others) or ignoring mathematical reality.
Signal vs. Noise
Data Curation
45–80 %
Self-estimated share of data scientists' working time (surveys 2016–2020) spent on data acquisition, data preparation, cleaning, and curation
Training Tokens
15 T+
Trillions of tokens (Llama 3: over 15, Qwen3: about 36 trillion) from internet sources form the training basis of modern language models — but the internet is not a representative sample
Data Principle
Quality > Quantity
A smaller model on curated data often outperforms a larger model on unfiltered data
Analogy: Studying with Mixed-Quality Notes
You have 1,000 pages of study material, but 700 pages contain outdated information, duplicated passages, contradictions, and irrelevant tangents. Only 300 pages contain accurate, well-structured content. A student who reads all 1,000 pages will be confused — they'll memorize contradictions and outdated facts alongside correct information, unable to distinguish which is which. A student who reads only the curated 300 pages will learn more effectively, despite seeing less material. Machine learning works identically: more data helps only when the additional data increases the signal, not the noise.
Example: Language Model Training
GPT-3 was trained on hundreds of billions of tokens scraped from the internet. The internet is not a representative sample of human knowledge — it is disproportionately English-language, Western-centric, and skewed toward tech-savvy demographics. Adding more data from the same source (more internet text) does not correct these distributions; it reinforces them. This is why GPT-3 already, and subsequent models even more so, invested heavily in data curation: filtering, deduplication, quality scoring, and deliberate weighting of sources. Since Chinchilla (2022) at the latest, it has been clear: progress does not come from "more parameters" alone – just as important is more data per parameter, and that data only helps if it is carefully curated.
ImageNet: The Dataset That Changed Everything
The creation of the dataset that made Deep Learning possible. In 2009, Fei-Fei Li and her team presented the ImageNet paper, introducing a visual database that would transform computer vision — at launch it contained around 3.2 million hand-annotated images in approximately 5,200 categories. Expanded to its full size, ImageNet later comprised over 14 million hand-annotated images and around 22,000 categories, based on WordNet hierarchies, addressing the critical bottleneck: the shortage of large, high-quality training data. Annotation was carried out throughout the project by around 49,000 workers from 167 countries via Amazon Mechanical Turk — an unprecedentedly collaborative effort. What began as a poster in a corner of a Miami Beach convention center grew into the annual ImageNet Challenge (ILSVRC) and became one of the three drivers of modern AI development. ImageNet enabled AlexNet's 2012 breakthrough and laid the foundation for autonomous vehicles, facial recognition, and medical imaging.
Caution
Misconception: "More Data Always Fixes Bias"
If the data source itself is skewed, scaling up amplifies rather than corrects the skew. A dataset of 1 million biased examples produces a more confidently biased model than a dataset of 10,000 biased examples. The fix is not volume but representativeness: ensuring the training distribution matches the intended deployment distribution across all relevant dimensions.
Gebru et al. (2021) proposed a standardized documentation format for datasets — comparable to datasheets for electronic components. Every dataset should document: How was the data collected? Which populations are represented? What are the known limitations? What is the dataset suitable for — and what is it not suitable for? These "Datasheets for Datasets" establish professional data governance and make quality assessment of training data systematic and traceable. For high-risk AI, this becomes mandatory: the EU AI Act (Art. 10) requires training data that is sufficiently representative and examined for possible biases – after the postponement by the Digital Omnibus, this applies to stand-alone high-risk systems from 2 December 2027.
Interactive: What Type of Bias Is This?
You have learned about three error sources: biased training data, conflicting fairness metrics, and poor data quality. Use the diagnosis tree to systematically determine which mechanism is at play in a given scenario.
Note
Key Takeaways
- Bias is a data problem: AI models don't generate their own judgments — they compress the patterns of their training data. Biased data produces biased predictions.
- Perfect fairness is mathematically impossible: The Impossibility Theorem proves that three intuitive fairness metrics cannot all be satisfied when base rates differ.
- Data quality beats data quantity: More data helps only when it increases the signal. Scaling data from a biased source amplifies the bias.
- Bias is not a bug but a constraint: Professional AI development means identifying biases, consciously choosing tradeoffs, and investing in data curation.
Check
Quiz: Bias & Data Quality
A credit scoring model trained on historical lending decisions reproduces the biases of past loan officers. Which type of training bias does this primarily represent?
Select one answer
1. A credit scoring model trained on historical lending decisions reproduces the biases of past loan officers. Which type of training bias does this primarily represent?
2. A disease affects 10% of Population X and 2% of Population Y. A screening algorithm is tuned so that it stays calibrated and both populations have an equal false negative rate. What is the unavoidable consequence according to the Impossibility Theorem?
3. A company doubles its training dataset by scraping more text from the same web sources. Why might the model's bias get worse rather than better?
4. An AI startup claims their hiring algorithm has "zero bias" because it achieves perfect calibration across all demographic groups. Based on the Impossibility Theorem, what can you conclude?
5. Which statement best describes the relationship between data quantity and data quality in ML training?
Check
Checkpoint: Do you understand Bias & Data Quality?
Can you explain this in your own words? Tick what you've got.
0 of 3 done