Back to Learning Area

Tensors: The Language of GPUs

Tensors: the term that sounds more intimidating than the thing actually is.

  • Fundamentals
  • 12 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How computers process data

What is a tensor?

A single number is called a scalar. Scalar means: a single value. Many numbers in a row are called a vector. A grid of numbers is a matrix. Many matrices stacked are called a tensor.

Everything must be a tensor

A GPU is a fast calculation chip. GPU means: graphics processing unit. A GPU performs many calculations at once. It only understands tensors. The computer converts images into tensors. It converts text into tensors.

Batches: many at once

A GPU has thousands of calculation cores. Calculation cores are small calculators in the chip. A single image leaves most cores empty. That is why you send many images at once. This is called a batch. Batch means: many examples together.

You already know scalars, vectors, and matrices. But deep learning does not stop at 2D. A color image needs three layers (Red, Green, Blue) — that is a 3D tensor. A batch of images adds another dimension — 4D. A batch of videos? 5D.

In this article, you will climb the dimension ladder, understand why EVERY AI input gets converted to a tensor, and learn why GPUs demand batches of data rather than one example at a time.

The Dimension Ladder

Definition

Tensor

Analogy

Think of an Excel workbook. A scalar is a single cell (A1 = 42). A vector is a column (A1:A10). A matrix is a worksheet. A 3D tensor is the entire workbook with multiple stacked sheets — like R, G, B color channels. A 4D tensor? A folder of workbooks.

Example

Excel sheets can have different sizes — in a tensor, every "sheet" must be exactly the same size. And Excel can only calculate across multiple sheets with special formulas — tensors are designed exactly for that.
  1. 1. ScalarA single value. No indices needed. Shape: () — e.g. temperature = 23.5
  2. 2. VectorA row of values. One index needed. Shape: (n) — e.g. weights = [0.3, 0.7, 0.1]
  3. 3. MatrixA grid of values. Two indices. Shape: (m, n) — e.g. grayscale image (4, 4) = 16 values
  4. 4. 3D TensorStacked matrices. Three indices. Shape: (k, m, n) — e.g. RGB image (3, 4, 4) = 48 values
  5. 5. 4D TensorStack of 3D tensors. Shape: (b, k, m, n) — e.g. batch of 32 images (32, 3, 256, 256)

A tiny 4x4 RGB image shows the dimension ladder in action. Grayscale: a matrix with shape (4, 4) = 16 values. Color: a 3D tensor with shape (3, 4, 4) = 48 values. Three layers stacked: Red, Green, Blue. Pixel at position (2, 3): Red=220, Green=45, Blue=180 — a vivid magenta.

import numpy as np

# Scalar (0D)
scalar = np.array(42)          # Shape: ()

# Vector (1D)
vector = np.array([1.0, 2.0, 3.0])  # Shape: (3,)

# Matrix (2D)
matrix = np.zeros((4, 4))      # Shape: (4, 4)

# 3D Tensor (RGB image)
rgb_image = np.random.randint(0, 256, (3, 4, 4))  # Shape: (3, 4, 4)

print(rgb_image.shape)  # (3, 4, 4)
print(rgb_image[0, 2, 3])  # Red value at position (2,3)

Caution

Misconception: Tensor in Code = Tensor in Physics

In physics, a tensor has specific transformation rules under coordinate changes (e.g., stress tensor, metric tensor). In PyTorch and TensorFlow, a tensor is simply a multi-dimensional array with GPU support and automatic gradient tracking. Same name, different meaning.

A 0D scalar holds a single value. But each new dimension lets you represent more: a vector with 768 entries (“768-dimensional” in embedding jargon, yet a 1D tensor) encodes the meaning of an entire word, a matrix transforms that meaning — and a 4D tensor can push 32 images through a neural network simultaneously.

Interactive: Three Views of Tensors

Tensors are abstract — but different analogies make them tangible. Switch between three perspectives to understand what a tensor is: a data cube for stacking, a mathematical transformation, or a numbered storage system.

What is a Tensor?

A tensor is the fundamental data structure in modern AI systems. Depending on the perspective, tensors can be understood in very different ways. Switch between three analogies to experience different perspectives.

Tensor as a Multi-Dimensional Data Cube

Think of a tensor as a container that organizes numbers across multiple dimensions. A scalar is a single point. A vector is a line of numbers. A matrix is a sheet with rows and columns. A 3D tensor is a cube. And it goes further: a 4D tensor is a shelf full of cubes.

Scalar (Rank 0): 42
Vector (Rank 1): [3, 7, 1, 9]
Matrix (Rank 2): [[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]
Tensor (Rank 3): [[[1,2],[3,4]],
[[5,6],[7,8]]]
= 2 Layers x 2 Rows x 2 Columns
Concrete Example

A color image of 256 x 256 pixels is stored as a rank-3 tensor: [3, 256, 256] — three color channels (red, green, blue), height, and width. A batch of 32 images creates a rank-4 tensor: [32, 3, 256, 256].

Strength of this Analogy

Makes the structure tangible: How many dimensions does the tensor have? What shape does it have?

Limitation of this Analogy

Doesn't explain why tensors are useful for computation — only how they store data.

Tensor Dimensions Compared

RankMathematicsData CubeAI Example
0ScalarSingle pointLearning rate: 0.001
1VectorLine of numbersWord Embedding: [768]
2MatrixSheet with rows + columnsWeight matrix: [768, 512]
33D TensorCube of numbersColor image: [3, 256, 256]
44D TensorShelf full of cubesImage batch: [32, 3, 256, 256]
Active Analogy:1 / 3Exploring all perspectives helps with understanding

Why ML Thinks in Tensors

GPUs and TPUs process dense, fixed-size numerical arrays with vectorized instructions. Therefore every input — image, text, audio, tabular data — must be "translated" into a tensor with a defined shape. The GPU speaks only one language: tensor.

Image (channels × height × width)

(3, 256, 256 )

Text (5 tokens × 768 numbers)

(5, 768 )

Audio (128 time steps × 80 frequency bands)

(128, 80 )

Table (1000 rows × 10 columns)

(1000, 10 )

Together with the convention (such as channels first), the shape tells you what each dimension means. Wrong shape = wrong interpretation = bugs or nonsensical results. The shape of a tensor is the single most important piece of metadata in deep learning.

Caution

Shape Errors: A Perennial Problem in Deep Learning

Shape mismatch errors are among the most common runtime errors in deep learning. You will inevitably encounter them:

RuntimeError: mat1 and mat2 shapes cannot be multiplied
  (32x768 and 512x256)

# Expected: (32, 512) @ (512, 256) -> (32, 256)
# Got: (32, 768) @ (512, 256) -> Error!

Tip: Print the shape at every intermediate step. print(tensor.shape) is your best friend when debugging.

Why fixed size? A tensor is rectangular: along each axis, every entry has the same length. So the examples in a batch can only be stacked into one tensor if they are the same length. Variable-length inputs get padded to a fixed maximum length or truncated.

PyTorch uses Channels First: (Batch, Channels, Height, Width) = (B, C, H, W). TensorFlow uses Channels Last by default: (Batch, Height, Width, Channels) = (B, H, W, C). Same pixel values, completely different memory layout. When porting between frameworks, you must explicitly permute — otherwise the network interprets channels as pixels and vice versa.

# PyTorch: Channels First (B, C, H, W)
pytorch_image = torch.randn(1, 3, 256, 256)

# TensorFlow: Channels Last (B, H, W, C)
tf_image = tf.random.normal([1, 256, 256, 3])

# Reorder into TensorFlow layout (still a PyTorch tensor)
tf_image = pytorch_image.permute(0, 2, 3, 1)
# (1, 3, 256, 256) -> (1, 256, 256, 3)

Both are n-dimensional arrays with a shape. The difference: tensors can run on the GPU and support automatic differentiation (autograd). NumPy arrays run on the CPU only and have no gradient tracking. In practice, you often convert between the two: NumPy for data preparation, tensors for training.

import numpy as np
import torch

# NumPy -> Tensor (on GPU)
np_array = np.array([[1.0, 2.0], [3.0, 4.0]])
tensor = torch.from_numpy(np_array).cuda()

# Tensor -> NumPy (back to CPU)
np_back = tensor.cpu().numpy()

# Tensor has gradient tracking
w = torch.tensor([2.0], requires_grad=True)
loss = (w * 3 - 6) ** 2
loss.backward()  # Gradient computed automatically
print(w.grad)    # tensor([0.])

Batching: Feeding the GPU

GPUs have thousands of cores all executing the same operation on different data simultaneously (SIMD — Single Instruction, Multiple Data). Processing one image at a time wastes most cores. That is why the batch dimension is prepended.

1 RGB image (256x256): 3 x 256 x 256 x 4 bytes

786 KB

Batch of 32 images: 32 x 786 KB

~25 MB

RTX 4090 VRAM: hundreds of images at once (plus model and intermediate results)

24 GB

Three Reasons for Batching

GPU utilization: All cores busy simultaneously. A single image uses only a fraction of the available compute power.

Statistical stability: The gradient averaged over a batch is less noisy than the gradient of a single example. Training converges more smoothly.

Trade-off: Larger batch = better GPU utilization, but more memory usage. Batch size is a critical hyperparameter.

Shape Progression

Image (3, 256, 256) → Batch (32, 3, 256, 256) → Video-Batch (32, 30, 3, 256, 256)
Each new dimension has clear meaning
import torch

# Single image: shape (3, 256, 256)
img = torch.randn(3, 256, 256)

# Batch of 32 images: shape (32, 3, 256, 256)
batch = torch.stack([torch.randn(3, 256, 256) for _ in range(32)])
print(batch.shape)  # torch.Size([32, 3, 256, 256])

# Memory: 32 * 3 * 256 * 256 * 4 bytes = ~25 MB

Caution

Misconception: Larger Batch = Always Better

More is not always better. Batch size 1 wastes GPU capacity — but batches that are too large cause out-of-memory errors. Additionally, very large batches can hurt generalization: the model converges faster but tends to land in sharp minima that transfer poorly to new data.

Note

Key Takeaways

  • A tensor is an n-dimensional array with a fixed shape. Scalar = 0D, vector = 1D, matrix = 2D, and beyond. The shape tuple tells you exactly how large each dimension is.
  • Every AI input — image, text, audio — must be encoded as a tensor of fixed shape before a GPU can process it. Shape mismatches are among the most common bugs in deep learning.
  • GPUs process batches, not individual examples. The batch dimension is prepended, turning a 3D image tensor into a 4D batch tensor — so thousands of cores can work in parallel.

Check

Quiz: Tensors

Not completed

What is a tensor?

Select one answer

Answer Key: 1) B · 2) C · 3) D · 4) B

Check

Learning Goals

Can you explain this in your own words? Tick what you've got.

0 of 3 done