The Network

What the middle layers of a neural network learn — and can't explain to anyone.

Architectures 14 min Intermediate June 1, 2026

A single neuron can draw a dividing line. But the world is not made of straight lines — it's made of faces in photos, meaning in text, and patterns in sound. To go from one neuron to intelligent pattern recognition, you need layers of neurons that work together, each building on the last.

These Hidden Layers are where the actual computation happens — invisible from the outside, but essential for everything a neural network can do.

Hidden Layer

AnalogyDefinition
Imagine a factory. Raw materials arrive at the entrance (Input Layer). At each station inside, workers transform the material — cutting, shaping, painting. These stations are the Hidden Layers. You never see the intermediate products from outside. At the end, the finished product rolls out (Output Layer). Each worker's skill (weight) is refined through practice (training).

Analogy:

Imagine a factory. Raw materials arrive at the entrance (Input Layer). At each station inside, workers transform the material — cutting, shaping, painting. These stations are the Hidden Layers. You never see the intermediate products from outside. At the end, the finished product rolls out (Output Layer). Each worker's skill (weight) is refined through practice (training).

Definition:

A Hidden Layer is a layer of neurons between Input and Output whose intermediate results are never directly visible. Each neuron computes z = Wx + b and applies an activation function. In a Dense (Fully Connected) Layer, every neuron connects to every neuron in the next layer.

From Neuron to Network

A neural network organizes neurons into layers. The Input Layer receives the raw data — no computation happens here. The Hidden Layers perform the actual processing. The Output Layer delivers the final result.

MNIST Network Architecture

Handwriting recognition: 784 pixels → 10 digits

Input Layer 784 neurons (28×28 pixels)
784
Hidden Layer 1 128 neurons (Dense)
128
Hidden Layer 2 64 neurons (Dense)
64
Output Layer 10 neurons (digits 0-9)
10

As soon as a network has more than one Hidden Layer, it's considered "deep" — hence the name Deep Learning. Modern architectures push this depth to extremes: GPT-4 consists of roughly 120 Transformer blocks, functioning as an enormous sequence of Hidden Layers.

784
Input Neurons 28×28 pixels of an MNIST image
~109,000
Parameters Trainable weights and biases
~120
Transformer Blocks GPT-4 architecture (estimate)

Where do the ~109,000 parameters come from? Each connection between two neurons has a weight, and each neuron has a bias:

  • Layer 1 → 2: 784 × 128 weights + 128 biases = 100,480
  • Layer 2 → 3: 128 × 64 weights + 64 biases = 8,256
  • Layer 3 → Output: 64 × 10 weights + 10 biases = 650

Total: 100,480 + 8,256 + 650 = 109,386 parameters

The Forward Pass

How does data flow through the network? The Forward Pass is the computation step where input data is passed layer by layer until a prediction emerges at the end.

Step 1: Linear Combination
z = W · x + b
Step 2: Activation
a = ReLU(z) = max(0, z)
Entire Network
output = f3(f2(f1(input)))

At each layer, the same thing happens: First, the input vector is multiplied by the weight matrix and the bias is added (z = Wx + b). Don't panic: a vector is just a list of numbers, a matrix is a table of numbers. Then an activation function is applied (a = ReLU(z)). The output of one layer becomes the input of the next — mathematically, a function composition.

Unlike a relay race where the baton is passed unchanged, the data is actively transformed at each handoff.

Batch Processing

In practice, individual data points are never processed alone — entire batches are. Hundreds of examples flow through the network simultaneously. Since this is essentially matrix multiplication, GPUs can parallelize these computations extremely efficiently.

This is how you define this Forward Pass in PyTorch. Calling netz(batch) sends the data through all layers:

netz = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 10)
)

Feature Hierarchies — What Hidden Layers Learn

The fascinating question: What do the millions of parameters actually store? Deep networks automatically learn so-called feature hierarchies — they decompose the world into different levels of abstraction.

1
Pixels Raw input data: color values and brightness
2
Edges Layer 1 detects edges and simple textures
3
Shapes Middle layers combine edges into geometric patterns
4
Objects Deep layers recognize complex concepts: faces, dog breeds, buildings

Zeiler and Fergus (2014) visualized CNN layers and demonstrated exactly this: Layer 1 detects edges in all directions. Layer 2 responds to textures and patterns. Layer 5 identifies highly specific concepts like faces or church towers. These visualizations partially debunk the myth that Hidden Layers are a "black box."

The decisive advantage over classical Machine Learning: Instead of painstakingly defining features by hand (Feature Engineering), the network discovers relevant features automatically.

Interactive: Explore Network Architecture

Click on the individual layers of the MNIST network and observe how the neuron count shrinks from 784 input pixels through the hidden layers to 10 output classes. Note the enormous connection counts between layers — they explain why training a network is so computationally intensive.

⋯784⋯128⋯64⋯10

Click on a layer to see details.

"Deeper Is Always Better" — Not So Fast

A common misconception: More layers mean a better model. The reality is more complex.

Shallow (1 Hidden Layer)

Fast to train. Few parameters. Good for tabular data and small datasets. Random Forest and SVM are often better.

Deep (Many Hidden Layers)

Learns complex patterns. Needs lots of data and compute. Risk: Vanishing Gradient and Overfitting.

Wrong Assumption: "More Layers = Better Model"

Networks that are too deep suffer from two problems: First, the Vanishing Gradient — gradients shrink through many layers during training, causing early layers to stop learning. Second, Overfitting — too many parameters memorize training data instead of generalizing. For many tasks with tabular data, classical models like Random Forest or SVM are the better choice.

Key Takeaways

  1. A neural network has three layer types: Input (raw data), Hidden (computation), Output (result). Hidden Layers are named so because their outputs are never directly visible.
  2. "Deep" means more than one Hidden Layer. Modern networks have hundreds of layers (GPT-4: ~120 Transformer blocks).
  3. The Forward Pass repeats at each layer: matrix multiplication (z = Wx + b), then activation (a = ReLU(z)). The output of one layer becomes the input of the next.
  4. Hidden Layers automatically learn feature hierarchies: edges → shapes → objects. This replaces manual Feature Engineering.
  5. More depth is not always better — Vanishing Gradient and Overfitting set limits. For many problems, simpler models are more appropriate.

Quiz: Hidden Layers

Question 1 / 5
Not completed

Why are Hidden Layers called "hidden"?

Select one answer

1. Why are Hidden Layers called "hidden"?

  • ☐ A) Because their code is encrypted and cannot be read
  • ☐ B) Because their intermediate outputs are never directly visible — only the final result of the Output Layer is seen
  • ☐ C) Because they are removed after training to make the model faster
  • ☐ D) Because they use a special "hidden" activation function

2. A network has the architecture: 100 inputs → 50 hidden neurons → 10 outputs. All layers are Dense. How many trainable weight parameters does it have (excluding biases)?

  • ☐ A) 160 (100 + 50 + 10)
  • ☐ B) 1,500 (100 × 50 / 2 — only half the connections)
  • ☐ C) 5,500 (100 × 50 + 50 × 10)
  • ☐ D) 50,000 (100 × 50 × 10)

3. You are building a model to predict house prices from 5 numerical features (size, rooms, age, location, condition). You have 500 training examples. Which approach is more appropriate?

  • ☐ A) A deep network with 10 Hidden Layers — more layers always capture more complex patterns
  • ☐ B) A shallow model like Linear Regression or Random Forest — the dataset is small and tabular, so a deep network would likely overfit
  • ☐ C) A deep network with Dropout — Dropout solves all overfitting problems regardless of dataset size
  • ☐ D) No model needed — 5 features can be analyzed by eye

4. A colleague claims: "Hidden Layers are a complete black box — we have absolutely no idea what happens inside them." Evaluate this statement.

  • ☐ A) Correct — Hidden Layer activations are mathematically impossible to interpret
  • ☐ B) Correct — we can see the weights but they are random numbers with no meaning
  • ☐ C) Incorrect — techniques like Zeiler and Fergus (2014) visualization show that we CAN see what layers learn: early layers detect edges, deeper layers recognize objects
  • ☐ D) Incorrect — Hidden Layers are fully transparent because we can read the source code

5. In a CNN for image recognition, Layer 1 detects edges and Layer 5 recognizes faces. What would happen if you removed Layers 2-4 and connected Layer 1 directly to Layer 5?

  • ☐ A) The network would still recognize faces, just slightly less accurately
  • ☐ B) The network would fail — Layer 5 expects the intermediate representations (shapes, textures) that Layers 2-4 build from edges. Without the hierarchy, edges alone cannot compose into faces.
  • ☐ C) The network would be faster because it has fewer layers to compute
  • ☐ D) The network would automatically compensate by learning edges and faces in Layer 1
Answer Key: 1) B · 2) C · 3) B · 4) C · 5) B

Checkpoint: Hidden Layers

  • Why are Hidden Layers actually called 'hidden', and what exactly is a Dense Layer?
  • How would you explain the Forward Pass to someone with no technical background?
  • Why does a network fail when you skip layers in the feature hierarchy?

Article

Article Intermediate
The Spark: Activation Functions

The one small kink in the line without which neural networks wouldn't learn anything at all.

10min
Article Advanced
The Heart of Learning

How a neural network sends its errors home — and makes everything along the way a little better.

12min
Article Intermediate
The Artificial Neuron

Frank Rosenblatt's 1958 idea that suddenly became relevant again 60 years later.

10min
Article Intermediate
The AI Winters

Why artificial intelligence was written off multiple times without caring.

12min
Article Intermediate
The Chain Rule: Differentiating Nested Functions

A high school math rule that made the AI revolution possible in the first place.

10min
Article Advanced
Computer Vision (CNNs): How Machines Learned to See

How machines learned to read images and could suddenly tell dogs from cats — most of the time.

12min
Article Intermediate
The Path to the Valley: Gradient Descent

How gradient descent finds the lowest point in a landscape with millions of hills — most of the time.

10min
Article Beginner
The Machine's Knobs — Parameters vs. Hyperparameters

The fine line between "learned by the machine" and "guessed by you".

8min
Article Beginner
Linear & Logistic Regression

The mathematical foundation that every deep learning course only gets to after three hours.

8min
Article Intermediate
How AI Measures Its Mistakes: Loss Functions

Loss functions are a model's pain threshold — without them, no learning.

11min
Article Intermediate
Data Tables & Transformations (Matrices)

The 2D grid of numbers where every ML library primarily thinks.

12min
Article Beginner
When the Model Memorizes (Overfitting)

How to notice that the model didn't learn but memorized.

7min
Article Intermediate
Partial Derivatives & The Gradient

The gradient: the vector that every optimization follows — downhill, hopefully.

13min
Article Advanced
Time & Sequences (RNNs)

The architecture that gave AI memory — before someone found a better way.

14min
Article Intermediate
Tensors: The Language of GPUs

Tensors: the term that sounds more intimidating than the thing actually is.

12min
Article Intermediate
Transfer Learning & Fine-Tuning

How to win over an AI model instead of raising it from scratch — invaluable from the start.

10min
Article Advanced
The Breakthrough — Transformers & Attention

What "Attention Is All You Need" (2017) actually changed — and why the world hasn't gone back.

12min

Demo

Demo Intermediate
GANs (Generative Models)

Interactive demonstration of Generative Adversarial Networks

30min
Demo Intermediate
Neuroevolution

Watch virtual cars learn to drive using neural networks and genetic algorithms - or take the wheel yourself and challenge the AI

30min
Demo Intermediate
Neural Network Playground

Click layers and neurons together, choose a dataset and activation function, and watch the network learn to separate the data in real time.

35min
Demo Beginner
Perceptron (Neural Networks)

Discover the first artificial neuron - the Big Bang of machine learning from 1957.

20min
Demo Beginner
Supervised Learning

Join Sharlock Helmes in his cleverest case: learning to distinguish between genuine clues and Moriarty's sophisticated red herrings. Elementary, my dear algorithm!

25min

Glossary

Hidden LayersPredictionParameterReLUOverfittingRandom ForestGPTTransformer ArchitectureBiasConvolutional Neural Network (CNN)Deep LearningDropoutActivation FunctionBackpropagationFeedforward NetworkFully Connected LayerMultilayer PerceptronNeural NetworkUniversal Approximation TheoremVanishing GradientWeight

Timeline

AlexNet SuccessBackpropagation AlgorithmLeNet and the Birth of CNNsDeep Belief Networks: The Renaissance of Deep LearningThe Deep Learning RevolutionDropout RegularizationPerceptron: The First Learning Neural NetworkResNet: Residual Networks Transform Deep LearningTransformer: 'Attention Is All You Need'Universal Approximation Theorem