The Machine's Knobs — Parameters vs. Hyperparameters
The fine line between "learned by the machine" and "guessed by you".
Fundamentals 8 min Beginner May 18, 2026
When headlines announce that GPT-4 has "trillions of parameters," most people imagine a developer sitting at a console, manually adjusting trillions of knobs. The reality is the exact opposite: those parameters are values the machine figured out entirely on its own. The actual knobs the developer turns — the hyperparameters — number in the dozens at most.
Confusing these two categories is the most common beginner mistake in machine learning. In this article, you'll learn what the difference is, why it matters, and how to systematically find the right settings.
Parameters — The Machine's Internal Wiring
Parameter
AnalogyDefinition
Imagine the internal circuitry of a high-end audio system during auto-calibration. Hundreds of resistors, capacitors, and amplifier stages adjust themselves automatically when the system runs its calibration routine. You never open the case or touch these components — they are the internal result of the machine's optimization process.
Analogy:
Imagine the internal circuitry of a high-end audio system during auto-calibration. Hundreds of resistors, capacitors, and amplifier stages adjust themselves automatically when the system runs its calibration routine. You never open the case or touch these components — they are the internal result of the machine's optimization process.
Definition:
Parameters are the internal numerical values that an ML model discovers and adjusts by itself during training, typically using algorithms like gradient descent (a mathematical search method that incrementally finds the best values). The most common parameters are weights and biases. In a simple linear model y = mx + b, both m and b are parameters. After training, parameters are frozen and used for predictions. The total number of parameters defines a model's capacity — Llama 3 has 70 billion of them.
The hi-fi analogy breaks at three points: an audio system has hundreds of components — Llama 3 has 70 billion parameters. Calibration takes seconds — LLM training takes months. A technician could theoretically open the system and adjust components manually — the parameter space of a neural network is so high-dimensional that any manual intervention is impossible.
Concrete Example: Spam Filter
A spam filter using logistic regression with 5 features (exclamation marks, links, capitalization ratio, sender reputation, word count). The model automatically learns 5 weights (one per feature) plus 1 bias = exactly 6 parameters. After training on 1,000 emails, the weight for exclamation marks might be 0.8 (strong spam signal), sender reputation -0.3 (negative = trusted sender), and the bias -0.5. These 6 numbers ARE the trained model.
Common Misconception
"The billions of parameters mentioned in the news are settings the developer chose." — Wrong. They are weights the model learned by itself. A developer of GPT-4 chose perhaps 50 to 100 hyperparameters. The trillions of parameters were found by the training algorithm over months of computation.
Hyperparameters — The Developer's Control Panel
Hyperparameter
AnalogyDefinition
The external knobs and dials on a radio — volume, bass, treble, reception mode. The engineer adjusts these before listening. The radio then operates within these settings. You can change the knobs and restart, but the radio will never turn its own knobs.
Analogy:
The external knobs and dials on a radio — volume, bass, treble, reception mode. The engineer adjusts these before listening. The radio then operates within these settings. You can change the knobs and restart, but the radio will never turn its own knobs.
Definition:
Hyperparameters are the algorithmic settings that must be fixed before training begins. They control HOW the model learns, not WHAT it learns. They remain constant during an entire training run. The most important hyperparameters are: (1) Learning Rate — how large each optimization step is. (2) Epochs — how many times the model sees the entire training dataset. (3) Batch Size — how many data points are processed before each parameter update.
This analogy also breaks: a radio has about 5 knobs with immediately audible effects — an ML model has dozens of hyperparameters with complex, often counter-intuitive interactions. On a radio you hear the result instantly — after changing a hyperparameter you wait hours or days for the training run.
The learning rate determines the step size of the optimization. Too high: the model overshoots the optimum and loses all sense of direction. Too low: training takes forever and gets stuck in a good-but-not-best valley (a so-called local minimum). Epochs determine how many times the model processes all training data. Batch size determines how many data points are processed per update step.
Experiment: Same Data, Different Settings
Run
Learning Rate
Epochs
Result
1
0,1
10
LR 0.01 · 10 epochs → 85% — Solid baseline run
2
0,001
10
LR 0.0001 · 10 epochs → 72% — Learning rate too low, model barely learns
3
0,1
100
LR 0.01 · 50 epochs → 92% — More passes find deeper patterns
4
10,0
10
LR 10.0 · 10 epochs → 50% — Far too high, model lost all sense of direction (50% in spam-or-not means blind guessing)
Common Misconception
"More epochs always produce a better model." Wrong — beyond a certain point, the model starts memorizing the training data instead of finding generalizable patterns. That's overfitting (topic of the next article). Equally wrong: "A higher learning rate means faster, better learning." Too-high values cause the model to overshoot the optimum — it loses all sense of direction.
Parameters vs. Hyperparameters — Inside vs. Outside
Parameters (internal)
Learned by the model itself (weights, biases). Billions in large models. Automatically optimized via gradient descent. Cannot be set manually. Analogy: the internal circuitry of a hi-fi system.
Hyperparameters (external)
Set by the developer before training (learning rate, epochs, batch size). Dozens, not billions. Must be found manually or by search. Analogy: the external knobs on a radio.
Finding the Right Settings
There is no formula that computes the perfect hyperparameters. ML engineers spend a large fraction of their working time systematically trying different combinations — hyperparameter tuning.
1
Define search space Which hyperparameters should be tested? What value ranges make sense?
2
Choose strategy Grid Search (all combinations), Random Search (random samples), or AutoML (automated)
3
Run experiments Train each candidate and evaluate on a validation dataset
4
Compare results Weigh accuracy, training time, and resource usage against each other
5
Select best configuration Do a final training run with the best hyperparameters and deploy
Grid Search defines a fixed grid of values and tests every combination. Guaranteed to find the best option within the grid — but expensive.
Random Search draws random combinations from the search space. Bergstra and Bengio showed in 2012 that Random Search is often more efficient than Grid Search, especially when only a few hyperparameters actually matter.
AutoML goes further: a secondary ML system searches for the best hyperparameters of the primary model. AI optimizing AI — meta-learning. Google prominently introduced AutoML in 2018.
A fitting analogy: a chef developing a new recipe. He doesn't know if the cake bakes better at 180°C or 200°C, or whether 30 or 45 minutes is the right baking time. Grid Search: systematically try all combinations. Random Search: test random temperature-time pairs. AutoML: hire an autonomous sous-chef.
Random Search 8 random trials — Best result: 93.8% — nearly as good in less than half the time
Common Misconception
"Just test every possible combination to guarantee the best model." — In theory, yes. In practice, impossible. Even a modest setup with 5 values for each of 6 hyperparameters yields 5⁶ = 15,625 experiments. At 2 hours each, that's 3.6 years of compute time.
Interactive: Why Grid Search Explodes
Grid Search tests all combinations. With n values per hyperparameter, the number of experiments grows with the number of hyperparameters: 1 HP = n trials, 2 HPs = n², 3 HPs = n³. Move the slider and observe how quickly costs explode — and why Random Search or Bayesian Optimization (O(log n)) become attractive.
220
O(log n)6.6
O(n)100
O(n²)10.000
O(n³)1.000.000
Moderate Input
At n=100, the difference becomes visible: O(n²) requires 10.000 operations, while O(n) needs only 100. O(log n) needs just 6.6 — that's 15x less than O(n).
Ratio to O(n)
Complexity
Operations
Factor vs. O(n)
O(log n)
6.6
15x faster
O(n)
100
1x (Reference)
O(n²)
10.000
100x slower
O(n³)
1.000.000
10000x slower
Summary
Key Takeaways
Parameters are learned by the model during training (weights, biases) — the developer never touches them. When someone says a model has "70 billion parameters," they mean learned weights.
Hyperparameters are configured by the developer before training (learning rate, epochs, batch size). They control HOW the model learns, not WHAT it learns.
Finding good hyperparameters is trial-and-error, not calculation. Grid Search tests all combinations; Random Search samples randomly and is often faster. AutoML automates the search entirely.
Quiz: Parameters vs. Hyperparameters
Question 1 / 4
Not completed
A neural network has 1,000 weights and 50 biases. How many parameters does it have?
1. A neural network has 1,000 weights and 50 biases. How many parameters does it have?
☐ A) 50
☐ B) 1,000
☐ C) 1,050
☐ D) It depends on the learning rate
2. A developer trains a spam filter with learning rate 0.1 and achieves 85% accuracy. She then retrains with learning rate 10.0 and achieves only 50%. What most likely happened?
☐ A) The model needed more epochs
☐ B) The model overshot the optimal values and diverged
☐ C) The training data was too small
☐ D) The batch size was wrong
3. A researcher has 4 hyperparameters, each with 5 candidate values. She uses Grid Search. How many experiments must she run?
☐ A) 20
☐ B) 25
☐ C) 125
☐ D) 625
4. Two teams train the same network architecture on the same dataset. Team A achieves 94% accuracy; Team B achieves 72%. Architecture and data are identical. What is the most likely explanation?
☐ A) Team A has a faster computer
☐ B) Team A used better hyperparameters
☐ C) Team B's model has fewer parameters
☐ D) The random initialization always determines the final result
Answer Key: 1) C · 2) B · 3) D · 4) B
Checkpoint
Learning Goals
When someone says Llama 3 has 70 billion parameters — are those developer settings or values the model learned on its own?
What happens when you set the learning rate extremely high, and why is the opposite (extremely low) also problematic?
Why doesn't a developer simply test every possible hyperparameter combination via Grid Search to guarantee the best model?