Tensors: The Language of GPUs
Tensors: the term that sounds more intimidating than the thing actually is.
You already know scalars, vectors, and matrices. But deep learning does not stop at 2D. A color image needs three layers (Red, Green, Blue) — that is a 3D tensor. A batch of images adds another dimension — 4D. A batch of videos? 5D.
In this article, you will climb the dimension ladder, understand why EVERY AI input gets converted to a tensor, and learn why GPUs demand batches of data rather than one example at a time.
The Dimension Ladder
Definition
Tensor
Analogy
Example
Definition
- 1. ScalarA single value. No indices needed. Shape: () — e.g. temperature = 23.5
- 2. VectorA row of values. One index needed. Shape: (n) — e.g. weights = [0.3, 0.7, 0.1]
- 3. MatrixA grid of values. Two indices. Shape: (m, n) — e.g. grayscale image (4, 4) = 16 values
- 4. 3D TensorStacked matrices. Three indices. Shape: (k, m, n) — e.g. RGB image (3, 4, 4) = 48 values
- 5. 4D TensorStack of 3D tensors. Shape: (b, k, m, n) — e.g. batch of 32 images (32, 3, 256, 256)
A tiny 4x4 RGB image shows the dimension ladder in action. Grayscale: a matrix with shape (4, 4) = 16 values. Color: a 3D tensor with shape (3, 4, 4) = 48 values. Three layers stacked: Red, Green, Blue. Pixel at position (2, 3): Red=220, Green=45, Blue=180 — a vivid magenta.
import numpy as np
# Scalar (0D)
scalar = np.array(42) # Shape: ()
# Vector (1D)
vector = np.array([1.0, 2.0, 3.0]) # Shape: (3,)
# Matrix (2D)
matrix = np.zeros((4, 4)) # Shape: (4, 4)
# 3D Tensor (RGB image)
rgb_image = np.random.randint(0, 256, (3, 4, 4)) # Shape: (3, 4, 4)
print(rgb_image.shape) # (3, 4, 4)
print(rgb_image[0, 2, 3]) # Red value at position (2,3)Caution
Misconception: Tensor in Code = Tensor in Physics
In physics, a tensor has specific transformation rules under coordinate changes (e.g., stress tensor, metric tensor). In PyTorch and TensorFlow, a tensor is simply a multi-dimensional array with GPU support and automatic gradient tracking. Same name, different meaning.
A 0D scalar holds a single value. But each new dimension lets you represent more: a vector with 768 entries (“768-dimensional” in embedding jargon, yet a 1D tensor) encodes the meaning of an entire word, a matrix transforms that meaning — and a 4D tensor can push 32 images through a neural network simultaneously.
Interactive: Three Views of Tensors
Tensors are abstract — but different analogies make them tangible. Switch between three perspectives to understand what a tensor is: a data cube for stacking, a mathematical transformation, or a numbered storage system.
What is a Tensor?
A tensor is the fundamental data structure in modern AI systems. Depending on the perspective, tensors can be understood in very different ways. Switch between three analogies to experience different perspectives.
Tensor as a Multi-Dimensional Data Cube
Think of a tensor as a container that organizes numbers across multiple dimensions. A scalar is a single point. A vector is a line of numbers. A matrix is a sheet with rows and columns. A 3D tensor is a cube. And it goes further: a 4D tensor is a shelf full of cubes.
A color image of 256 x 256 pixels is stored as a rank-3 tensor: [3, 256, 256] — three color channels (red, green, blue), height, and width. A batch of 32 images creates a rank-4 tensor: [32, 3, 256, 256].
Tensor Dimensions Compared
| Rank | Mathematics | Data Cube | AI Example |
|---|---|---|---|
| 0 | Scalar | Single point | Learning rate: 0.001 |
| 1 | Vector | Line of numbers | Word Embedding: [768] |
| 2 | Matrix | Sheet with rows + columns | Weight matrix: [768, 512] |
| 3 | 3D Tensor | Cube of numbers | Color image: [3, 256, 256] |
| 4 | 4D Tensor | Shelf full of cubes | Image batch: [32, 3, 256, 256] |
Why ML Thinks in Tensors
GPUs and TPUs process dense, fixed-size numerical arrays with vectorized instructions. Therefore every input — image, text, audio, tabular data — must be "translated" into a tensor with a defined shape. The GPU speaks only one language: tensor.
Image (channels × height × width)
(3, 256, 256 )
Text (5 tokens × 768 numbers)
(5, 768 )
Audio (128 time steps × 80 frequency bands)
(128, 80 )
Table (1000 rows × 10 columns)
(1000, 10 )
Together with the convention (such as channels first), the shape tells you what each dimension means. Wrong shape = wrong interpretation = bugs or nonsensical results. The shape of a tensor is the single most important piece of metadata in deep learning.
Caution
Shape Errors: A Perennial Problem in Deep Learning
Shape mismatch errors are among the most common runtime errors in deep learning. You will inevitably encounter them:
RuntimeError: mat1 and mat2 shapes cannot be multiplied
(32x768 and 512x256)
# Expected: (32, 512) @ (512, 256) -> (32, 256)
# Got: (32, 768) @ (512, 256) -> Error!Tip: Print the shape at every intermediate step. print(tensor.shape) is your best friend when debugging.
Why fixed size? A tensor is rectangular: along each axis, every entry has the same length. So the examples in a batch can only be stacked into one tensor if they are the same length. Variable-length inputs get padded to a fixed maximum length or truncated.
PyTorch uses Channels First: (Batch, Channels, Height, Width) = (B, C, H, W). TensorFlow uses Channels Last by default: (Batch, Height, Width, Channels) = (B, H, W, C). Same pixel values, completely different memory layout. When porting between frameworks, you must explicitly permute — otherwise the network interprets channels as pixels and vice versa.
# PyTorch: Channels First (B, C, H, W)
pytorch_image = torch.randn(1, 3, 256, 256)
# TensorFlow: Channels Last (B, H, W, C)
tf_image = tf.random.normal([1, 256, 256, 3])
# Reorder into TensorFlow layout (still a PyTorch tensor)
tf_image = pytorch_image.permute(0, 2, 3, 1)
# (1, 3, 256, 256) -> (1, 256, 256, 3)Both are n-dimensional arrays with a shape. The difference: tensors can run on the GPU and support automatic differentiation (autograd). NumPy arrays run on the CPU only and have no gradient tracking. In practice, you often convert between the two: NumPy for data preparation, tensors for training.
import numpy as np
import torch
# NumPy -> Tensor (on GPU)
np_array = np.array([[1.0, 2.0], [3.0, 4.0]])
tensor = torch.from_numpy(np_array).cuda()
# Tensor -> NumPy (back to CPU)
np_back = tensor.cpu().numpy()
# Tensor has gradient tracking
w = torch.tensor([2.0], requires_grad=True)
loss = (w * 3 - 6) ** 2
loss.backward() # Gradient computed automatically
print(w.grad) # tensor([0.])Batching: Feeding the GPU
GPUs have thousands of cores all executing the same operation on different data simultaneously (SIMD — Single Instruction, Multiple Data). Processing one image at a time wastes most cores. That is why the batch dimension is prepended.
1 RGB image (256x256): 3 x 256 x 256 x 4 bytes
786 KB
Batch of 32 images: 32 x 786 KB
~25 MB
RTX 4090 VRAM: hundreds of images at once (plus model and intermediate results)
24 GB
Three Reasons for Batching
GPU utilization: All cores busy simultaneously. A single image uses only a fraction of the available compute power.
Statistical stability: The gradient averaged over a batch is less noisy than the gradient of a single example. Training converges more smoothly.
Trade-off: Larger batch = better GPU utilization, but more memory usage. Batch size is a critical hyperparameter.
Shape Progression
import torch
# Single image: shape (3, 256, 256)
img = torch.randn(3, 256, 256)
# Batch of 32 images: shape (32, 3, 256, 256)
batch = torch.stack([torch.randn(3, 256, 256) for _ in range(32)])
print(batch.shape) # torch.Size([32, 3, 256, 256])
# Memory: 32 * 3 * 256 * 256 * 4 bytes = ~25 MBCaution
Misconception: Larger Batch = Always Better
More is not always better. Batch size 1 wastes GPU capacity — but batches that are too large cause out-of-memory errors. Additionally, very large batches can hurt generalization: the model converges faster but tends to land in sharp minima that transfer poorly to new data.
Note
Key Takeaways
- A tensor is an n-dimensional array with a fixed shape. Scalar = 0D, vector = 1D, matrix = 2D, and beyond. The shape tuple tells you exactly how large each dimension is.
- Every AI input — image, text, audio — must be encoded as a tensor of fixed shape before a GPU can process it. Shape mismatches are among the most common bugs in deep learning.
- GPUs process batches, not individual examples. The batch dimension is prepended, turning a 3D image tensor into a 4D batch tensor — so thousands of cores can work in parallel.
Check
Quiz: Tensors
What is a tensor?
Select one answer
1. What is a tensor?
2. An RGB image has 64x64 pixels. What is the tensor shape (channels first)?
3. You process a batch of 8 RGB images at 128x128 pixels. What is the tensor shape?
4. Why does a GPU process data in batches instead of one at a time?
Check
Learning Goals
Can you explain this in your own words? Tick what you've got.
0 of 3 done