Unsupervised Learning
Learning without an answer key — the more demanding but often more practical variant of machine learning.
Your streaming service groups movies into categories that no one taught it. Your bank blocks fraudulent transactions in milliseconds — even if no one ever showed the system this exact fraud pattern. How do machines discover hidden patterns in data when there are no answers to learn from?
After the previous articles on Supervised Learning, where models train on labeled data, we flip the approach. What happens when there are no labels, no correct answers, no teacher? Unsupervised Learning is how machines discover hidden structure in raw, unlabeled data — from customer segmentation to fraud detection. This article introduces three foundational techniques.
The Paradigm Without a Teacher
In Supervised Learning, the model has answers to learn from — the labels. In Unsupervised Learning, the algorithm must find patterns entirely on its own. This is not easier — it is fundamentally different.
Three pillars support this paradigm: clustering (finding groups), dimensionality reduction (finding the essence), and anomaly detection (finding outliers). Each technique tackles a different facet of the challenge, and each comes with trade-offs that the practitioner — not the machine — must navigate.
Word2Vec: Words as Vectors
The transformation of word representation through semantic vector spaces. On January 16, 2013, Tomas Mikolov and his Google team published the influential paper 'Efficient Estimation of Word Representations in Vector Space'. Word2Vec transformed NLP by representing words as dense, low-dimensional vectors (typically 100 to 300 dimensions) that capture semantic and syntactic relationships — a break from the large, sparse one-hot vectors used in earlier methods. The two architectural variants, CBOW (Continuous Bag of Words) and Skip-Gram, learned from large text corpora that similar words appear in similar contexts. The famous example demonstrated vector arithmetic: king - man + woman ≈ queen. With tens of thousands of citations, Mikolov's work became one of the most influential NLP papers. Word2Vec laid the foundation for all modern embedding techniques and enabled semantic reasoning in vector spaces. This innovation paved the way for transformer architectures and modern large language models.
Clustering with k-Means
Definition
k-Means Clustering
Analogy
Definition
The Four-Step Algorithm
k-Means works in four clearly defined, repeating steps:
- 1. Choose k random starting positions as cluster centerscentroids
- 2. Assign each data point to the nearest centroid
- 3. Recalculate centroids as the mean of all assigned points
- 4. Repeat steps 2 and 3 until convergence is reached
Worked Example: Customer Segmentation
An online retailer analyzes 10,000 customers using three features: age, income, and purchase frequency. With k=4, k-Means discovers four customer profiles that were never provided as labels: budget-conscious students, young families with high purchase frequency, well-earning professionals, and affluent retirees. These groups emerge solely from geometric proximity in three-dimensional feature space — the algorithm has zero prior knowledge about life stages or purchasing behavior. Names like "budget-conscious students" are only assigned afterwards, by a person looking at the groups.
The k Problem: Who Decides the Number?
The biggest catch with k-Means: the machine cannot determine on its own how many natural groups exist in the data. You, the practitioner, must set the hyperparameter k in advance. Choose k too low, and distinct groups get merged. Choose k too high, and natural clusters get artificially split. The choice of k is a human decision, not a machine discovery.
A proven approach for selecting k is the Elbow Method. You compute the sum of squared distances from each point to its cluster center (Within-Cluster Sum of Squares, WCSS) for various k values — say from k=1 to k=10. Plotting WCSS against k typically produces a curve that drops steeply at first and then flattens out. The inflection point — the "elbow" — marks where additional clusters bring only marginal improvement. Imagine: at k=3 the curve drops sharply, at k=4 only slightly, at k=5 barely at all. Then k=3 or k=4 is a sensible choice. But be careful: the method is a heuristic, not a guarantee. Sometimes there is no clear bend, and you need domain knowledge.
Caution
Common Misconception
Myth: k-Means always finds the correct clusters.
Fact: The result heavily depends on the choice of k and the random initialization of centroids. Different starting positions can produce entirely different cluster assignments. The Elbow Method helps with choosing k but is a heuristic — not a guarantee. In practice, k-Means is therefore run several times from different starting points, keeping the result with the smallest sum of squared distances (WCSS); k-means++ also spreads out the starting points.
Interactive: k-Means step by step
Here k-Means runs on the practical example above: each point is a customer, placed by age (to the right) and income (upward) — two of the three features, so it fits on the screen. The large circles are the centroids; k = 4 is preset as in the example. Click “Next step” to walk through the four steps. Then try another k or “Other start points”: not every start ends in the same clusters.
Interactive: Hierarchical clustering & cut height
k-Means needs the number of clusters k up front. Hierarchical (agglomerative) clustering does not: it merges the points bottom-up into a tree — and only the height at which you cut that tree decides how many clusters come out. Try it.
Dimensionality Reduction with PCA
Definition
Principal Component Analysis (PCA)
Analogy
Definition
Connection to Eigenvectors
PCA builds on the vectors and matrices from Path I.C. The covariance matrix records how strongly each pair of features varies together. An eigenvector of this matrix is a direction that the matrix only stretches, never rotates — the stretch factor is called the eigenvalue. The algorithm computes the eigenvectors of the dataset's covariance matrix. These eigenvectors define the new axes in the space — the principal components. The corresponding eigenvalues indicate how much variance each axis captures. You keep the eigenvectors with the largest eigenvalues and discard the rest.
Dimensionality Reduction
784 → 50
MNIST: 784 pixel features compressed to ~50 principal components (94% reduction), while preserving digit distinguishability
Worked Example: MNIST Digits
The MNIST dataset contains 70,000 handwritten digits, each with 28×28 = 784 pixels. Many border pixels are almost always empty (pure background) — they carry no relevant information. PCA identifies that just about 50 principal components already capture about 82% of the variation (for 95% you would need around 150). The digits remain distinguishable, but the number of features drops by 94% – storage and computation time for subsequent classifiers drop considerably.
Why is the eigenvector with the largest eigenvalue the first principal component? Because the eigenvalues of the covariance matrix directly measure the variance along the corresponding eigenvector direction. The largest eigenvalue = the direction with the greatest data spread. The second principal component is orthogonal (perpendicular) to the first — it captures the second-largest spread not explained by the first. Because the axes are eigenvectors of the covariance matrix, the data's values along them are also uncorrelated: no information is double-counted. Linear algebra tells us: eigenvectors of a symmetric matrix can always be chosen orthogonal to each other — this is precisely why PCA works so elegantly.
Caution
Common Misconception
Myth: PCA reduces dimensions without losing information.
Fact: With real data, PCA practically always discards variance. The dropped components contained real data — just less of it than the ones you kept. The question is never "Do we lose information?" but "How much loss is acceptable for how much efficiency gain?" And if about 8% of the variance is missing, that does not mean 8% of the data points are missing: every point is kept, but each one loses its share along the discarded directions – and for individual points, that share can be exactly what matters.
Anomaly Detection
Definition
Anomaly Detection
Analogy
Definition
Two Approaches Compared
Measures how far a point is from known clusters. Strength: Works well with spherical clusters. Weakness: Struggles with unevenly distributed data.
Measures how sparsely populated a point's neighborhood is. Strength: Detects anomalies even in irregularly shaped data. Weakness: Sensitive to the choice of neighborhood radius.
Class Distribution
99.9 %
In credit card fraud, more than 99.9% of all transactions are legitimate — the needle-in-a-haystack challenge
Worked Example: Credit Card Fraud
A credit card company monitors transactions. More than 99.9% are legitimate. The model learns each customer's purchase profile: German stores, 10–200 EUR range, daytime purchases. Then three rapid 5,000 EUR transactions come from a foreign country. These points are geometrically far from the learned profile and get flagged automatically — not because the system "knows" fraud, but because the transactions don't match the normal pattern.
Caution
Common Misconception
Myth: Unsupervised Learning is easier because you don't need labels.
Fact: The absence of labels makes evaluation extremely difficult. In Supervised Learning, you have ground truth to measure accuracy against. In anomaly detection, you often can't tell whether a flagged point is a true anomaly or a false positive — without human investigation.
Memory: Which Technique Matches Which Application?
Test your knowledge of unsupervised learning techniques. Find the matching pairs: each technique has a typical application. How quickly can you find all six pairs? DBSCAN, autoencoders and t-SNE do not appear in the article – the hints on the cards will help you.
Note
Key Takeaways
- k-Means groups data into k clusters — but k itself is a human decision, not a machine discovery.
- PCA compresses data by keeping the axes of greatest variance — deliberately trading some information for massive efficiency gains.
- Anomaly detection learns what "normal" looks like and flags everything that deviates — a paradigm that powers fraud detection, quality control, and cybersecurity.
Check
Knowledge Check: Unsupervised Learning
What must a practitioner specify before running k-Means that the algorithm cannot determine on its own?
Select one answer
1. What must a practitioner specify before running k-Means that the algorithm cannot determine on its own?
2. You run k-Means on a customer dataset with k=5, but the Elbow Method plot shows a clear bend at k=3. What is the most appropriate next step?
3. A dataset has 200 features. After PCA, you keep 15 principal components that explain 92% of the total variance. A colleague says: "We only lost 8% of our data." Why is this statement misleading?
4. An anomaly detection system monitoring network traffic flags 1,000 events per day. After investigation, 990 are false positives and 10 are real intrusions. Without the system, all 10 intrusions would go undetected. Is the system useful?
5. You apply PCA to the MNIST digit dataset (784 features per image) and reduce to 50 components. You then train a classifier on the reduced data. Which of the following is a direct benefit of this PCA step?
6. A credit card company uses anomaly detection. A new customer makes a single large purchase on their first day. The system flags it as anomalous. Why might this be a false positive?
Check
Checkpoint: Unsupervised Learning
Can you explain this in your own words? Tick what you've got.
0 of 3 done