Data Tables & Transformations (Matrices)
The 2D grid of numbers where every ML library primarily thinks.
You already know vectors — ordered lists of numbers that represent data points. But what happens when you stack many vectors together? You get a matrix — a rectangular grid of numbers. Every table of numbers, every dataset, every neural network layer is built on matrices.
In this article, you will learn what matrices are, how to multiply them (and why the definition is not arbitrary), and why this single operation is the computational heart of modern AI.
What Is a Matrix?
Definition
Matrix
Analogy
Example
Definition
Rows = data points, columns = features. A 1000×5 matrix stores 1000 houses with 5 properties each.
The matrix defines HOW an input vector is turned into an output vector. Weight matrices in neural networks are transformations.
Three houses with the features area, rooms, and year built form a 3×3 matrix. Each row is a house, each column is a feature.
import numpy as np
# Three houses as a 3x3 matrix
houses = np.array([
[120, 4, 1990],
[85, 3, 2005],
[200, 6, 1975]
])
print(houses.shape) # (3, 3)
print(houses[1, 2]) # 2005 — year built of house 2Caution
Misconception: Matrices Are Just Tables
Matrices are far more than data storage. Their real power lies in representing transformations. A weight matrix in a neural network does not store data — it defines HOW inputs are transformed into outputs.
Every tabular dataset in machine learning is a matrix: rows are examples, columns are features. And every fully connected network layer is a matrix transformation.
Matrix Multiplication — Why This Definition?
Matrix multiplication looks complicated at first — but it has a deep reason: It exactly describes the composition of two transformations. Apply matrix B first, then matrix A, and the result is the matrix AB.
- 1. Take row i of matrix A.
- 2. Take column j of matrix B.
- 3. Compute the dot product (multiply element by element and add up).
- 4. The result goes at position (i, j) of the result matrix C.
Each element c_ij of the result is the dot product of row i of A with column j of B. The connection to the vectors article: matrix multiplication consists of many dot products.
The inner dimensions must match. An (m×n) matrix times an (n×p) matrix produces an (m×p) matrix. If the inner dimensions do not match, multiplication is undefined.
import numpy as np
# Weight matrix W (2x3) times input vector x (shape (3,))
W = np.array([[0.5, -0.3, 0.8],
[0.1, 0.7, -0.2]])
x = np.array([1, 2, 3])
print(W @ x) # [2.3, 0.9]
print(W.shape, '@', x.shape, '→', (W @ x).shape)
# (2, 3) @ (3,) → (2,)Caution
Misconception: A × B = B × A
Matrix multiplication is NOT commutative. A × B generally gives a different result from B × A. Often, only one of the two products is even defined because the dimensions do not match. Order matters!
In NumPy, * is the elementwise product (Hadamard) and @ is matrix multiplication. They give completely different results! The most common beginner mistake: using * when @ is intended.
import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A * B) # [[ 5, 12], [21, 32]] — elementwise
print(A @ B) # [[19, 22], [43, 50]] — matrix multiplicationInteractive: Row Times Column
The starting values are A and B from the deep dive on the Hadamard product: A @ B = [[19, 22], [43, 50]]. Change a number in A or B (whole numbers from −10 to 10), and the product updates instantly. Pick a cell of the result: the box marks the row of A and the column of B it comes from. The chart shows one of the matrices as a transformation: it reshapes the unit square.
Matrices in Neural Networks — Why GPUs Calculate
Every fully connected layer of a neural network computes exactly one matrix multiplication: weight matrix times input plus bias. After that, the layer usually applies an activation function to the result, number by number; the heavy lifting, though, is still the matrix multiplication. Training means adjusting the entries of the weight matrix to reduce the error.
Shape Chain Through a Mini-Network
Instead of processing one vector at a time, many inputs are stacked into a matrix. This way, an entire batch (e.g., 32 images) is processed in ONE matrix multiplication. This is dramatically faster. The inputs then sit as rows on the left: X @ W instead of W × x, as in the code below; the weight matrix has the shape (features in × features out).
A100 GPU (FP16, Tensor Cores)
312 TFLOPS
CPU (order of magnitude)
~1–5 TFLOPS
Speedup (peak performance)
~60–300 ×
Why are GPUs so fast at this? Because each dot product in a matrix multiplication can be computed completely independently of the others. A CPU has a few very fast cores (e.g., 16). A GPU has thousands of slower cores (the A100, for example, has 6,912, plus 432 Tensor Cores that each multiply a small matrix block per clock cycle). For matrices — where thousands of simple calculations happen simultaneously — the GPU is dramatically superior. NVIDIA's A100 datasheet gives the 312 TFLOPS for 16-bit numbers (FP16/BF16) on the Tensor Cores; the CPU figure is only an order of magnitude. The A100 came out in 2020; the newer NVIDIA B200 from the Blackwell generation reaches about 2,250 TFLOPS in the same number format.
import numpy as np
# Mini-network (without bias and activation): 5 data points, 3 features
X = np.random.randn(5, 3) # Input: (5, 3)
W1 = np.random.randn(3, 4) # Layer 1: (3, 4)
W2 = np.random.randn(4, 2) # Layer 2: (4, 2)
hidden = X @ W1 # (5, 3) @ (3, 4) = (5, 4)
output = hidden @ W2 # (5, 4) @ (4, 2) = (5, 2)
print(f'Input: {X.shape} → Hidden: {hidden.shape} → Output: {output.shape}')The inner dimension disappears at each multiplication — like canceling fractions. (5×3) @ (3×4) = (5×4): the 3 cancels out. Then (5×4) @ (4×2) = (5×2): the 4 cancels out. Data flows through the network layer by layer, and dimensions change in a controlled way.
# Shape chain visualized:
# (batch × features_in) @ (features_in × features_out) = (batch × features_out)
# 5 × 3 @ 3 × 4 = 5 × 4
# 5 × 4 @ 4 × 2 = 5 × 2
# The inner dimension must always match!Note
Key Takeaways
- A matrix is a rectangular grid of numbers — it can represent a dataset (rows = examples) or a transformation (input → output).
- Matrix multiplication consists of dot products between rows and columns. It is defined this way because it describes the composition of linear transformations — not an arbitrary convention.
- Every fully connected neural network layer computes output = W × input + bias — one matrix multiplication. GPUs excel at this because every output element can be computed independently.
Check
Quiz: Matrices
A dataset has 500 images, each described by 3 features. What shape does the corresponding matrix have?
Select one answer
1. A dataset has 500 images, each described by 3 features. What shape does the corresponding matrix have?
2. A weight matrix W has shape (4×3) and the input vector x has shape (3,). What shape does W @ x have?
3. In NumPy, A * B and A @ B give different results. Which operator is correct for a neural network layer computation?
4. Why are GPUs much faster than CPUs for matrix multiplication, even though CPUs have higher clock speeds?
Check
Checkpoint: Matrices
Can you explain this in your own words? Tick what you've got.
0 of 3 done