Back to Learning Area

The Network

What the middle layers of a neural network learn — and can't explain to anyone.

  • Architectures
  • 14 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Layers in a neural network

What are hidden layers?

A neural network is a learnable computer program. Neural networks have multiple layers. The middle layers are called hidden layers. Hidden layers means: hidden layers. The actual calculation happens there.

How deep learning works

Many middle layers together are called deep learning. Deep learning means: deep learning with many layers. Each layer learns something different. The first layer sees simple patterns. The last layer recognizes complex objects.

From pixel to face

The layers build on each other. The first layer recognizes lines and edges. The next layer recognizes shapes. Later, the network recognizes entire faces.

A single neuron can draw a dividing line. But the world is not made of straight lines — it's made of faces in photos, meaning in text, and patterns in sound. To go from one neuron to intelligent pattern recognition, you need layers of neurons that work together, each building on the last.

These Hidden Layers are where the actual computation happens — invisible from the outside, but essential for everything a neural network can do.

Definition

Hidden Layer

Analogy

Imagine a factory. Raw materials arrive at the entrance (Input Layer). At each station inside, workers transform the material — cutting, shaping, painting. These stations are the Hidden Layers. You never see the intermediate products from outside. At the end, the finished product rolls out (Output Layer). Each worker's skill (weight) is refined through practice (training).

From Neuron to Network

A neural network organizes neurons into layers. The Input Layer receives the raw data — no computation happens here. The Hidden Layers perform the actual processing. The Output Layer delivers the final result.

MNIST Network Architecture

Handwriting recognition: 784 pixels → 10 digits

Input Layer 784 neurons (28×28 pixels)
784
Hidden Layer 1 128 neurons (Dense)
128
Hidden Layer 2 64 neurons (Dense)
64
Output Layer 10 neurons (digits 0-9)
10

As soon as a network has more than one Hidden Layer, it's considered "deep" — hence the name Deep Learning. Large language models push this depth to extremes: according to Meta, its Llama 3.1 405B consists of 126 Transformer blocks, functioning as an enormous sequence of Hidden Layers.

Input Neurons

784

28×28 pixels of an MNIST image

Parameters

~109,000

Trainable weights and biases

Transformer Blocks

126

Llama 3.1 405B (Meta, 2024)

Where do the ~109,000 parameters come from? Each connection between two neurons has a weight, and each neuron has a bias:

  • Input → Hidden 1: 784 × 128 weights + 128 biases = 100,480
  • Hidden 1 → Hidden 2: 128 × 64 weights + 64 biases = 8,256
  • Hidden 2 → Output: 64 × 10 weights + 10 biases = 650

Total: 100,480 + 8,256 + 650 = 109,386 parameters

The Forward Pass

How does data flow through the network? The Forward Pass is the computation step where input data is passed layer by layer until a prediction emerges at the end.

z = W · x + b
Step 1: Linear Combination
a = ReLU(z) = max(0, z)
Step 2: Activation
output = f3(f2(f1(input)))
Entire Network

At each layer, the same thing happens: First, the input vector is multiplied by the weight matrix and the bias is added (z = Wx + b). Don't panic: a vector is just a list of numbers, a matrix is a table of numbers. Then an activation function is applied (a = ReLU(z)). The output of one layer becomes the input of the next — mathematically, a function composition.

Unlike a relay race where the baton is passed unchanged, the data is actively transformed at each handoff.

Batch Processing

In training, individual data points are practically never processed alone — entire batches are. Hundreds of examples flow through the network simultaneously. Since this is essentially matrix multiplication, GPUs can parallelize these computations extremely efficiently.

This is how you define this Forward Pass in PyTorch. Calling netz(batch) sends the data through all layers:

netz = nn.Sequential(
    nn.Flatten(),
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 10)
)

Feature Hierarchies — What Hidden Layers Learn

The fascinating question: What do all these parameters actually store? Deep networks automatically learn so-called feature hierarchies — they decompose the world into different levels of abstraction.

  1. 1. PixelsRaw input data: color values and brightness
  2. 2. EdgesLayer 1 detects edges and simple textures
  3. 3. ShapesMiddle layers combine edges into geometric patterns
  4. 4. ObjectsDeep layers recognize complex concepts: faces, dog breeds, buildings

Zeiler and Fergus (2014) visualized CNN layers and demonstrated exactly this: Layer 1 detects edges in all directions. Layer 2 responds to corners and edge combinations, Layer 3 to textures. Layer 4 responds to class-specific parts like dog faces, Layer 5 to whole objects like keyboards or dogs. These visualizations partially debunk the myth that Hidden Layers are a "black box." Today, research goes much further: mechanistic interpretability traces features and their connections even inside large language models. MIT Technology Review named it one of its ten breakthrough technologies of 2026.

The decisive advantage over classical Machine Learning: Instead of painstakingly defining features by hand (Feature Engineering), the network discovers relevant features automatically.

Interactive: Explore Network Architecture

Click on the individual layers of the MNIST network and observe how the neuron count shrinks from 784 input pixels through the hidden layers to 10 output classes. Note the enormous connection counts between layers — they explain why training a network is so computationally intensive.

⋯784⋯128⋯64⋯10

Click on a layer to see details.

"Deeper Is Always Better" — Not So Fast

A common misconception: More layers mean a better model. The reality is more complex.

Shallow (1 Hidden Layer)

Fast to train. Few parameters. Good for tabular data and small datasets. Random Forest and Gradient Boosting are often better.

Deep (Many Hidden Layers)

Learns complex patterns. Needs lots of data and compute. Risk: Vanishing Gradient and Overfitting.

Caution

Wrong Assumption: "More Layers = Better Model"

Networks that are too deep suffer from two problems: First, the Vanishing Gradient — gradients shrink through many layers during training, causing early layers to stop learning. Second, Overfitting — too many parameters memorize training data instead of generalizing. Remedies like ReLU and Residual Connections still make very deep networks trainable today — but depth only pays off when the data and the task need it. For many tasks with tabular data, tree-based models like Random Forest or Gradient Boosting are the better choice. Pretrained tabular models such as TabPFN (published in Nature in 2025) are an exception: this Transformer was pretrained on millions of synthetic tables and beats even tuned tree methods on small datasets (up to 10,000 rows). A deep network you train from scratch on a few hundred rows of your own, however, remains the worse choice.

Note

Key Takeaways

  1. A neural network has three layer types: Input (raw data), Hidden (computation), Output (result). Hidden Layers are named so because their outputs are never directly visible.
  2. "Deep" means more than one Hidden Layer. Large modern networks have hundreds of layers (Llama 3.1 405B: 126 Transformer blocks with several layers each).
  3. The Forward Pass repeats at each layer: matrix multiplication (z = Wx + b), then activation (a = ReLU(z)). The output of one layer becomes the input of the next.
  4. Hidden Layers automatically learn feature hierarchies: edges → shapes → objects. This replaces manual Feature Engineering.
  5. More depth is not always better — Vanishing Gradient and Overfitting set limits. For many problems, simpler models are more appropriate.

Check

Quiz: Hidden Layers

Not completed

Why are Hidden Layers called "hidden"?

Select one answer

Answer Key: 1) B · 2) C · 3) B · 4) C · 5) B

Check

Checkpoint: Hidden Layers

Can you explain this in your own words? Tick what you've got.

0 of 3 done