Linear & Logistic Regression
The mathematical foundation that every deep learning course only gets to after three hours.
Many machine-learning models start the same way: multiply inputs by weights, add a bias, and check how wrong the prediction is. Linear regression does this for continuous values, logistic regression does it for classes — and together they form the blueprint that almost every neural network still follows today.
In the previous article, you learned what supervised learning is: data with labels, features, and a target value. Now you meet the first models that actually do something with them. The pattern you learn here — weighted sum, error measurement, non-linear activation — is exactly the skeleton of a neural network.
The Prediction Machine — Linear Regression
Definition
Linear Regression
Analogy
Example
Definition
w is the weight (the slope of the line), x is the input value, and b is the bias (the y-intercept). With multiple inputs, the formula becomes ŷ = w₁·x₁ + w₂·x₂ + … + b — instead of a line, it describes a plane or hyperplane.
Example: House Prices
A simple dataset: 50 m² → €150,000, 80 m² → €240,000, 120 m² → €360,000. The best-fit line: ŷ = 3,000 · x.
For a 100 m² apartment: ŷ = 3,000 · 100 = €300,000.
Suppose the actual price is €280,000. The residual (error) is: 280,000 − 300,000 = −€20,000. The model overestimates the price by €20,000.
Caution
Misconception: Linear regression only works with one input
The univariate formula ŷ = w · x + b is the teaching entry point. Real models almost always use multiple features simultaneously (multivariate). A house-price model might consider square metres, year built, floor level, and distance to city centre — each feature gets its own weight.
Keeping Score — Mean Squared Error
Definition
Mean Squared Error (MSE)
Analogy
Example
Definition
Why square and average the errors? Three reasons: (1) Without squaring, positive and negative errors cancel out — an error of +10 and one of −10 would average to zero. (2) Large errors are punished disproportionately: an error of 10 gives 100, but an error of 50 gives 2,500 — not 5 times more, but 25 times. (3) The mean makes MSE comparable across datasets of different sizes.
Example: Three Houses, Three Errors
Three houses with prediction errors: +€10,000, −€5,000, and −€50,000.
Squared: 100M, 25M, and 2,500M. MSE = (100 + 25 + 2,500) / 3 = 875M.
The third error (−50,000) dominates the entire MSE — that is by design. Squaring ensures the model cannot ignore large outliers.
Caution
Misconception: MSE is just a number you look at after training
MSE (or related functions) is actually the navigation signal: the model adjusts its weights in the direction that shrinks this value. Without a loss function, the model has no compass — it would not know which direction to adjust its weights.
Imagine the squaring function as a round bowl: a ball always rolls smoothly and unambiguously to the lowest point — there is no doubt which way is downhill. Absolute values (|y − ŷ|) are like a V-shaped valley with a sharp kink at the bottom where the optimisation algorithm can stumble. That is why the smooth squaring function is preferred. Additionally, logistic regression uses Log Loss (Binary Cross-Entropy) rather than MSE as its loss function, because MSE produces too flat a signal with probabilities.
Interactive: Beat the Line
From Numbers to Decisions — Logistic Regression
Definition
Logistic Regression
Analogy
Example
Definition
This z is exactly the result of the weighted sum from the previous step — the bouncer’s internally calculated score. The sigmoid function squeezes this number into the range between 0 and 1. For large positive z it approaches 1, for large negative z it approaches 0, and at z = 0 it gives exactly 0.5. This turns an unbounded weighted sum into a probability.
Example: Spam Classifier
A spam classifier with three features: suspicious words, unknown sender, and urgency language.
Weighted sum z = 1.8. Sigmoid(1.8) ≈ 0.858.
Since 0.858 > 0.5, the model classifies the email as "Spam" with about 86% model-confidence. In a different use case, a hospital might lower the threshold to 0.3 to catch more at-risk patients — better to warn once too often than to miss a case.
Caution
Misconception: Logistic regression predicts numeric values like prices
The name comes from its statistical ancestry: logistic regression "regresses" on probabilities via the logit link. In practice, it is a classifier. Joseph Berkson coined the term "logit" in 1944 — the name "logistic regression" for such models only caught on later and has stuck despite the confusion.
Output: continuous number (e.g., price in €). Use case: Regression (predicting values). Loss function: MSE.
Output: probability between 0 and 1. Use case: Classification (assigning classes). Loss function: Log Loss.
Interactive: The Sigmoid Formula Step by Step
Click on each term of the sigmoid formula to understand its role. The example shows a concrete spam classification with z = 2.5.
Sigmoid Formula
The output of the sigmoid function: a probability between 0 and 1. When σ(z) > 0.5, the model classifies as positive (e.g., spam); when σ(z) < 0.5, as negative.
Concrete Example
Given: Spam detection z = w·x + b = 0.5·5 + 0 = 2.5
Computation: z = w·x + b (Linear score) , σ(z) = 1/(1+e⁻ᶻ) (Sigmoid function)
The Big Picture — Bridge to the Artificial Neuron
What you learned in this article is more than three separate concepts. It is a chain that exactly matches the structure and training of an artificial neuron:
- 1. Weighted SumInputs × Weights + Bias → a number (just like linear regression)
- 2. Activation FunctionPass the number through a non-linear function (e.g., sigmoid)
- 3. Loss SignalCompare prediction to true value and measure the error (e.g., MSE)
- 4. Weight UpdateAdjust weights in the direction that shrinks the loss
You already understand 80% of an artificial neuron — you just don't know it yet. In the next learning path (Deep Learning), you will meet the neuron formally — and recognise that it is exactly this chain, just with more layers.
Adrien-Marie Legendre published the method of least squares in 1805 — the mathematical foundation on which MSE rests. Francis Galton made "regression" famous as an analysis tool in the 1880s: he observed that extremely tall parents tended to have children of more average height — a "regression to the mean." The name stuck, even though today it broadly refers to predicting numeric values from inputs. Joseph Berkson introduced the logit in 1944 and built on it a regression model for probabilities that today is mostly used as a classifier — a naming confusion that persists to this day.
Note
Key Takeaways
- A linear model predicts by multiplying each input by a weight and adding a bias — the result is a straight line (or hyperplane).
- MSE measures prediction quality by averaging squared errors, deliberately punishing large misses harder than small ones.
- Logistic regression wraps the same weighted sum in a sigmoid so the output stays between 0 and 1 — turning a regression machine into a classifier.
- Weights + bias + activation function + loss = the recipe card for almost every neural network.
Check
Comprehension Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Linear & Logistic Regression
In the equation ŷ = w · x + b — what does the term b represent?
Select one answer
1. In the equation ŷ = w · x + b — what does the term b represent?
2. A linear model predicts house prices: ŷ = 2,500 · x + 10,000. What does it predict for a 60 m² apartment?
3. Three predictions have errors of +4, −2, and +6. What is the MSE?
4. A logistic regression model outputs σ(z) = 0.72. A hospital uses a threshold of 0.60 to flag at-risk patients. What classification results — and what happens if the threshold is lowered to 0.40?
5. Why does MSE square the errors instead of simply adding them up?
6. Your team needs to predict whether a customer will cancel their contract (yes/no). A colleague suggests linear regression. Why is logistic regression better suited?