The Machine's Knobs — Parameters vs. Hyperparameters

The fine line between "learned by the machine" and "guessed by you".

When headlines announce that GPT-4 has "trillions of parameters," most people imagine a developer sitting at a console, manually adjusting trillions of knobs. The reality is the exact opposite: those parameters are values the machine figured out entirely on its own. The actual knobs the developer turns — the hyperparameters — number in the dozens at most.

Confusing these two categories is the most common beginner mistake in machine learning. In this article, you'll learn what the difference is, why it matters, and how to systematically find the right settings.

Parameters — The Machine's Internal Wiring

Imagine the internal circuitry of a high-end audio system during auto-calibration. Hundreds of resistors, capacitors, and amplifier stages adjust themselves automatically when the system runs its calibration routine. You never open the case or touch these components — they are the internal result of the machine's optimization process.

Analogy:

Definition:

Parameters are the internal numerical values that an ML model discovers and adjusts by itself during training, typically using algorithms like gradient descent (a mathematical search method that incrementally finds the best values). The most common parameters are weights and biases. In a simple linear model y = mx + b, both m and b are parameters. After training, parameters are frozen and used for predictions. The total number of parameters defines a model's capacity — Llama 3 has 70 billion of them.

The hi-fi analogy breaks at three points: an audio system has hundreds of components — Llama 3 has 70 billion parameters. Calibration takes seconds — LLM training takes months. A technician could theoretically open the system and adjust components manually — the parameter space of a neural network is so high-dimensional that any manual intervention is impossible.

A spam filter using logistic regression with 5 features (exclamation marks, links, capitalization ratio, sender reputation, word count). The model automatically learns 5 weights (one per feature) plus 1 bias = exactly 6 parameters. After training on 1,000 emails, the weight for exclamation marks might be 0.8 (strong spam signal), sender reputation -0.3 (negative = trusted sender), and the bias -0.5. These 6 numbers ARE the trained model.

"The billions of parameters mentioned in the news are settings the developer chose." — Wrong. They are weights the model learned by itself. A developer of GPT-4 chose perhaps 50 to 100 hyperparameters. The trillions of parameters were found by the training algorithm over months of computation.

Hyperparameters — The Developer's Control Panel

The external knobs and dials on a radio — volume, bass, treble, reception mode. The engineer adjusts these before listening. The radio then operates within these settings. You can change the knobs and restart, but the radio will never turn its own knobs.

Analogy:

Definition:

Hyperparameters are the algorithmic settings that must be fixed before training begins. They control HOW the model learns, not WHAT it learns. They remain constant during an entire training run. The most important hyperparameters are: (1) Learning Rate — how large each optimization step is. (2) Epochs — how many times the model sees the entire training dataset. (3) Batch Size — how many data points are processed before each parameter update.

This analogy also breaks: a radio has about 5 knobs with immediately audible effects — an ML model has dozens of hyperparameters with complex, often counter-intuitive interactions. On a radio you hear the result instantly — after changing a hyperparameter you wait hours or days for the training run.

The learning rate determines the step size of the optimization. Too high: the model overshoots the optimum and loses all sense of direction. Too low: training takes forever and gets stuck in a good-but-not-best valley (a so-called local minimum). Epochs determine how many times the model processes all training data. Batch size determines how many data points are processed per update step.

Run	Learning Rate	Epochs	Result
1	0,1	10	LR 0.01 · 10 epochs → 85% — Solid baseline run
2	0,001	10	LR 0.0001 · 10 epochs → 72% — Learning rate too low, model barely learns
3	0,1	100	LR 0.01 · 50 epochs → 92% — More passes find deeper patterns
4	10,0	10	LR 10.0 · 10 epochs → 50% — Far too high, model lost all sense of direction (50% in spam-or-not means blind guessing)

"More epochs always produce a better model." Wrong — beyond a certain point, the model starts memorizing the training data instead of finding generalizable patterns. That's overfitting (topic of the next article). Equally wrong: "A higher learning rate means faster, better learning." Too-high values cause the model to overshoot the optimum — it loses all sense of direction.

Parameters vs. Hyperparameters — Inside vs. Outside

Parameters (internal)

Learned by the model itself (weights, biases). Billions in large models. Automatically optimized via gradient descent. Cannot be set manually. Analogy: the internal circuitry of a hi-fi system.

Hyperparameters (external)

Set by the developer before training (learning rate, epochs, batch size). Dozens, not billions. Must be found manually or by search. Analogy: the external knobs on a radio.

Finding the Right Settings

There is no formula that computes the perfect hyperparameters. ML engineers spend a large fraction of their working time systematically trying different combinations — hyperparameter tuning.

Define search space Which hyperparameters should be tested? What value ranges make sense?

Choose strategy Grid Search (all combinations), Random Search (random samples), or AutoML (automated)

Run experiments Train each candidate and evaluate on a validation dataset

Compare results Weigh accuracy, training time, and resource usage against each other

Select best configuration Do a final training run with the best hyperparameters and deploy

Grid Search defines a fixed grid of values and tests every combination. Guaranteed to find the best option within the grid — but expensive.

Random Search draws random combinations from the search space. Bergstra and Bengio showed in 2012 that Random Search is often more efficient than Grid Search, especially when only a few hyperparameters actually matter.

AutoML goes further: a secondary ML system searches for the best hyperparameters of the primary model. AI optimizing AI — meta-learning. Google prominently introduced AutoML in 2018.

A fitting analogy: a chef developing a new recipe. He doesn't know if the cake bakes better at 180°C or 200°C, or whether 30 or 45 minutes is the right baking time. Grid Search: systematically try all combinations. Random Search: test random temperature-time pairs. AutoML: hire an autonomous sous-chef.

18 exp. / 36 hrs

Grid Search 3 × 3 × 2 = 18 combinations — Best result: 94.2%

8 exp. / 16 hrs

Random Search 8 random trials — Best result: 93.8% — nearly as good in less than half the time

"Just test every possible combination to guarantee the best model." — In theory, yes. In practice, impossible. Even a modest setup with 5 values for each of 6 hyperparameters yields 5⁶ = 15,625 experiments. At 2 hours each, that's 3.6 years of compute time.

Interactive: Why Grid Search Explodes

Grid Search tests all combinations. With n values per hyperparameter, the number of experiments grows with the number of hyperparameters: 1 HP = n trials, 2 HPs = n², 3 HPs = n³. Move the slider and observe how quickly costs explode — and why Random Search or Bayesian Optimization (O(log n)) become attractive.

Input size n = 100

220

O(log n)6.6

O(n)100

O(n²)10.000

O(n³)1.000.000

Moderate Input

At n=100, the difference becomes visible: O(n²) requires 10.000 operations, while O(n) needs only 100. O(log n) needs just 6.6 — that's 15x less than O(n).

Ratio to O(n)

Complexity	Operations	Factor vs. O(n)
O(log n)	6.6	15x faster
O(n)	100	1x (Reference)
O(n²)	10.000	100x slower
O(n³)	1.000.000	10000x slower

Summary

Parameters are learned by the model during training (weights, biases) — the developer never touches them. When someone says a model has "70 billion parameters," they mean learned weights.
Hyperparameters are configured by the developer before training (learning rate, epochs, batch size). They control HOW the model learns, not WHAT it learns.
Finding good hyperparameters is trial-and-error, not calculation. Grid Search tests all combinations; Random Search samples randomly and is often faster. AutoML automates the search entirely.

A neural network has 1,000 weights and 50 biases. How many parameters does it have?

1,000

1,050

It depends on the learning rate

1. A neural network has 1,000 weights and 50 biases. How many parameters does it have?

☐ A) 50
☐ B) 1,000
☐ C) 1,050
☐ D) It depends on the learning rate

2. A developer trains a spam filter with learning rate 0.1 and achieves 85% accuracy. She then retrains with learning rate 10.0 and achieves only 50%. What most likely happened?

☐ A) The model needed more epochs
☐ B) The model overshot the optimal values and diverged
☐ C) The training data was too small
☐ D) The batch size was wrong

3. A researcher has 4 hyperparameters, each with 5 candidate values. She uses Grid Search. How many experiments must she run?

☐ A) 20
☐ B) 25
☐ C) 125
☐ D) 625

4. Two teams train the same network architecture on the same dataset. Team A achieves 94% accuracy; Team B achieves 72%. Architecture and data are identical. What is the most likely explanation?

☐ A) Team A has a faster computer
☐ B) Team A used better hyperparameters
☐ C) Team B's model has fewer parameters
☐ D) The random initialization always determines the final result

Answer Key: 1) C · 2) B · 3) D · 4) B

Checkpoint

Learning Goals

When someone says Llama 3 has 70 billion parameters — are those developer settings or values the model learned on its own?
What happens when you set the learning rate extremely high, and why is the opposite (extremely low) also problematic?
Why doesn't a developer simply test every possible hyperparameter combination via Grid Search to guarantee the best model?

The Machine's Knobs — Parameters vs. Hyperparameters

Parameters — The Machine's Internal Wiring

Parameter

Analogy:

Definition:

Concrete Example: Spam Filter

Common Misconception

Hyperparameters — The Developer's Control Panel

Hyperparameter

Analogy:

Definition:

Experiment: Same Data, Different Settings

Common Misconception

Parameters vs. Hyperparameters — Inside vs. Outside

Finding the Right Settings

Common Misconception

Interactive: Why Grid Search Explodes

Ratio to O(n)

Summary

Key Takeaways

Quiz: Parameters vs. Hyperparameters

A neural network has 1,000 weights and 50 biases. How many parameters does it have?

Checkpoint

Learning Goals

Parameters — The Machine's Internal Wiring

Parameter

Analogy:

Definition:

Concrete Example: Spam Filter

Common Misconception

Hyperparameters — The Developer's Control Panel

Hyperparameter

Analogy:

Definition:

Experiment: Same Data, Different Settings

Common Misconception

Parameters vs. Hyperparameters — Inside vs. Outside

Finding the Right Settings

Common Misconception

Interactive: Why Grid Search Explodes

Ratio to O(n)

Summary

Key Takeaways

Quiz: Parameters vs. Hyperparameters

A neural network has 1,000 weights and 50 biases. How many parameters does it have?

Checkpoint

Learning Goals

Related Content

Article

How Good Is Your Model? Metrics That Actually Matter

When the Model Memorizes (Overfitting)

Programming vs. Training

The Raw Material: Data Engineering for Machine Learning

The Spark: Activation Functions

The Heart of Learning

Control & Finetuning

The Path to the Valley: Gradient Descent

The Network

Linear & Logistic Regression

How AI Measures Its Mistakes: Loss Functions

Supervised Learning — Learning with a Teacher

Transfer Learning & Fine-Tuning

Demo

Evolution (Optimization)

Gradient Descent

K-Means Clustering

Neuroevolution

Perceptron (Neural Networks)

Q-Learning

Supervised Learning

Glossary

Timeline