The Network
What the middle layers of a neural network learn — and can't explain to anyone.
A single neuron can draw a dividing line. But the world is not made of straight lines — it's made of faces in photos, meaning in text, and patterns in sound. To go from one neuron to intelligent pattern recognition, you need layers of neurons that work together, each building on the last.
These Hidden Layers are where the actual computation happens — invisible from the outside, but essential for everything a neural network can do.
Definition
Hidden Layer
Analogy
Definition
From Neuron to Network
A neural network organizes neurons into layers. The Input Layer receives the raw data — no computation happens here. The Hidden Layers perform the actual processing. The Output Layer delivers the final result.
MNIST Network Architecture
Handwriting recognition: 784 pixels → 10 digits
As soon as a network has more than one Hidden Layer, it's considered "deep" — hence the name Deep Learning. Large language models push this depth to extremes: according to Meta, its Llama 3.1 405B consists of 126 Transformer blocks, functioning as an enormous sequence of Hidden Layers.
Deep Belief Networks: The Renaissance of Deep Learning
Geoffrey Hinton reshaped the AI world in 2006 with his important paper on Deep Belief Networks. After years of neural networks being out of favor, he showed how deep neural networks could be trained efficiently. His innovation: layer-by-layer pre-training with Restricted Boltzmann Machines (RBMs). This 'greedy' learning strategy solved the problem of weight initialization and made deep learning practically applicable. The method stacks RBMs on top of each other and trains each layer individually before refining the entire network. Hinton's work ended the years-long obscurity of neural networks and initiated their renaissance. By 2009, DBNs had already significantly reduced error rates in speech recognition. In 2012, Hinton's team won the ImageNet Challenge (ILSVRC) with AlexNet — a deep convolutional neural network that used GPU training, ReLU, and dropout, and was no longer reliant on the RBM pre-training of DBNs. AlexNet achieved a top-5 error rate of 15.3% compared to 26.2% for the second-best team — a notable improvement. This moment marks the rebirth of neural networks and the beginning of today's AI boom.
Input Neurons
784
28×28 pixels of an MNIST image
Parameters
~109,000
Trainable weights and biases
Transformer Blocks
126
Llama 3.1 405B (Meta, 2024)
Where do the ~109,000 parameters come from? Each connection between two neurons has a weight, and each neuron has a bias:
- Input → Hidden 1: 784 × 128 weights + 128 biases = 100,480
- Hidden 1 → Hidden 2: 128 × 64 weights + 64 biases = 8,256
- Hidden 2 → Output: 64 × 10 weights + 10 biases = 650
Total: 100,480 + 8,256 + 650 = 109,386 parameters
The Forward Pass
How does data flow through the network? The Forward Pass is the computation step where input data is passed layer by layer until a prediction emerges at the end.
At each layer, the same thing happens: First, the input vector is multiplied by the weight matrix and the bias is added (z = Wx + b). Don't panic: a vector is just a list of numbers, a matrix is a table of numbers. Then an activation function is applied (a = ReLU(z)). The output of one layer becomes the input of the next — mathematically, a function composition.
Unlike a relay race where the baton is passed unchanged, the data is actively transformed at each handoff.
Batch Processing
In training, individual data points are practically never processed alone — entire batches are. Hundreds of examples flow through the network simultaneously. Since this is essentially matrix multiplication, GPUs can parallelize these computations extremely efficiently.
This is how you define this Forward Pass in PyTorch. Calling netz(batch) sends the data through all layers:
netz = nn.Sequential(
nn.Flatten(),
nn.Linear(784, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10)
)Feature Hierarchies — What Hidden Layers Learn
The fascinating question: What do all these parameters actually store? Deep networks automatically learn so-called feature hierarchies — they decompose the world into different levels of abstraction.
- 1. PixelsRaw input data: color values and brightness
- 2. EdgesLayer 1 detects edges and simple textures
- 3. ShapesMiddle layers combine edges into geometric patterns
- 4. ObjectsDeep layers recognize complex concepts: faces, dog breeds, buildings
Zeiler and Fergus (2014) visualized CNN layers and demonstrated exactly this: Layer 1 detects edges in all directions. Layer 2 responds to corners and edge combinations, Layer 3 to textures. Layer 4 responds to class-specific parts like dog faces, Layer 5 to whole objects like keyboards or dogs. These visualizations partially debunk the myth that Hidden Layers are a "black box." Today, research goes much further: mechanistic interpretability traces features and their connections even inside large language models. MIT Technology Review named it one of its ten breakthrough technologies of 2026.
The decisive advantage over classical Machine Learning: Instead of painstakingly defining features by hand (Feature Engineering), the network discovers relevant features automatically.
Interactive: Explore Network Architecture
Click on the individual layers of the MNIST network and observe how the neuron count shrinks from 784 input pixels through the hidden layers to 10 output classes. Note the enormous connection counts between layers — they explain why training a network is so computationally intensive.
"Deeper Is Always Better" — Not So Fast
A common misconception: More layers mean a better model. The reality is more complex.
Fast to train. Few parameters. Good for tabular data and small datasets. Random Forest and Gradient Boosting are often better.
Learns complex patterns. Needs lots of data and compute. Risk: Vanishing Gradient and Overfitting.
Caution
Wrong Assumption: "More Layers = Better Model"
Networks that are too deep suffer from two problems: First, the Vanishing Gradient — gradients shrink through many layers during training, causing early layers to stop learning. Second, Overfitting — too many parameters memorize training data instead of generalizing. Remedies like ReLU and Residual Connections still make very deep networks trainable today — but depth only pays off when the data and the task need it. For many tasks with tabular data, tree-based models like Random Forest or Gradient Boosting are the better choice. Pretrained tabular models such as TabPFN (published in Nature in 2025) are an exception: this Transformer was pretrained on millions of synthetic tables and beats even tuned tree methods on small datasets (up to 10,000 rows). A deep network you train from scratch on a few hundred rows of your own, however, remains the worse choice.
Note
Key Takeaways
- A neural network has three layer types: Input (raw data), Hidden (computation), Output (result). Hidden Layers are named so because their outputs are never directly visible.
- "Deep" means more than one Hidden Layer. Large modern networks have hundreds of layers (Llama 3.1 405B: 126 Transformer blocks with several layers each).
- The Forward Pass repeats at each layer: matrix multiplication (z = Wx + b), then activation (a = ReLU(z)). The output of one layer becomes the input of the next.
- Hidden Layers automatically learn feature hierarchies: edges → shapes → objects. This replaces manual Feature Engineering.
- More depth is not always better — Vanishing Gradient and Overfitting set limits. For many problems, simpler models are more appropriate.
Check
Quiz: Hidden Layers
Why are Hidden Layers called "hidden"?
Select one answer
1. Why are Hidden Layers called "hidden"?
2. A network has the architecture: 100 inputs → 50 hidden neurons → 10 outputs. All layers are Dense. How many trainable weight parameters does it have (excluding biases)?
3. You are building a model to predict house prices from 5 numerical features (size, rooms, age, location, condition). You have 500 training examples. Which approach is more appropriate?
4. A colleague claims: "Hidden Layers are a complete black box — we have absolutely no idea what happens inside them." Evaluate this statement.
5. In a CNN for image recognition, Layer 1 detects edges and Layer 5 recognizes faces. What would happen if you removed Layers 2-4 and connected Layer 1 directly to Layer 5, without retraining?
Check
Checkpoint: Hidden Layers
Can you explain this in your own words? Tick what you've got.
0 of 3 done