Back to Learning Area

Linear & Logistic Regression

The mathematical foundation that every deep learning course only gets to after three hours.

  • Fundamentals
  • 8 min
  • Beginner

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How AI makes predictions

Linear regression

Linear regression is a method. The program predicts numbers. An example: how expensive is a house? The program calculates the price based on the area. It uses weights. A weight determines how important an input value is.

Logistic regression

Logistic regression does not predict numbers. It sorts things into groups. A group is called a class. An example: spam or no spam. The result is a probability between 0 and 1. Over 0.5 means: the email is spam.

The basis of AI

These methods are the basis of AI. Every neural network uses them. Neural network means: an AI program with many layers. It calculates inputs with weights. Then it checks the error. Then it improves the weights.

Many machine-learning models start the same way: multiply inputs by weights, add a bias, and check how wrong the prediction is. Linear regression does this for continuous values, logistic regression does it for classes — and together they form the blueprint that almost every neural network still follows today.

In the previous article, you learned what supervised learning is: data with labels, features, and a target value. Now you meet the first models that actually do something with them. The pattern you learn here — weighted sum, error measurement, non-linear activation — is exactly the skeleton of a neural network.

The Prediction Machine — Linear Regression

Definition

Linear Regression

Analogy

Imagine a real-estate agent who, after hundreds of deals, has developed a rough pricing formula: "3,000 euros per square metre." This formula condenses experience into a straight line. That is exactly what linear regression does: it finds the line that best fits the data.

Example

The agent uses unquantifiable context (neighbourhood feel, renovation state) that they cannot put into numbers. The model can only use features it has been given as input — everything else is invisible.
ŷ = w · x + b
Formula: Linear Regression (univariate)

w is the weight (the slope of the line), x is the input value, and b is the bias (the y-intercept). With multiple inputs, the formula becomes ŷ = w₁·x₁ + w₂·x₂ + … + b — instead of a line, it describes a plane or hyperplane.

Example: House Prices

A simple dataset: 50 m² → €150,000, 80 m² → €240,000, 120 m² → €360,000. The best-fit line: ŷ = 3,000 · x.

For a 100 m² apartment: ŷ = 3,000 · 100 = €300,000.

Suppose the actual price is €280,000. The residual (error) is: 280,000 − 300,000 = −€20,000. The model overestimates the price by €20,000.

Caution

Misconception: Linear regression only works with one input

The univariate formula ŷ = w · x + b is the teaching entry point. Real models almost always use multiple features simultaneously (multivariate). A house-price model might consider square metres, year built, floor level, and distance to city centre — each feature gets its own weight.

Keeping Score — Mean Squared Error

Definition

Mean Squared Error (MSE)

Analogy

Imagine an archery target where the bull's eye is the true value and each arrow is a prediction. Arrows that land far from the centre count disproportionately because the "penalty area" (the square) grows with distance. An arrow that misses by twice as far does not cost twice as many penalty points — it costs four times as many.

Example

With outliers, a single wild arrow can dominate the entire score. In archery that is just bad luck, but with noisy data it can mislead the model.
MSE = (1/n) · ∑(yi − ŷi)²
Formula: Mean Squared Error

Why square and average the errors? Three reasons: (1) Without squaring, positive and negative errors cancel out — an error of +10 and one of −10 would average to zero. (2) Large errors are punished disproportionately: an error of 10 gives 100, but an error of 50 gives 2,500 — not 5 times more, but 25 times. (3) The mean makes MSE comparable across datasets of different sizes.

Example: Three Houses, Three Errors

Three houses with prediction errors: +€10,000, −€5,000, and −€50,000.

Squared: 100M, 25M, and 2,500M. MSE = (100 + 25 + 2,500) / 3 = 875M.

The third error (−50,000) dominates the entire MSE — that is by design. Squaring ensures the model cannot ignore large outliers.

Caution

Misconception: MSE is just a number you look at after training

MSE (or related functions) is actually the navigation signal: the model adjusts its weights in the direction that shrinks this value. Without a loss function, the model has no compass — it would not know which direction to adjust its weights.

Imagine the squaring function as a round bowl: a ball always rolls smoothly and unambiguously to the lowest point — there is no doubt which way is downhill. Absolute values (|y − ŷ|) are like a V-shaped valley with a sharp kink at the bottom where the optimisation algorithm can stumble. That is why the smooth squaring function is preferred. Additionally, logistic regression uses Log Loss (Binary Cross-Entropy) rather than MSE as its loss function, because MSE produces too flat a signal with probabilities.

Interactive: Beat the Line

From Numbers to Decisions — Logistic Regression

Definition

Logistic Regression

Analogy

Imagine a nightclub bouncer with an internal scoring system: appearance, behaviour, ID. He mentally calculates a score and converts it into a probability of entry. Unlike a real bouncer, the model has a precise, adjustable threshold that can be shifted for different contexts (e.g., more cautious for medical diagnoses).

Example

A real bouncer decides by gut feeling and experience — their threshold is fuzzy and context-dependent. The model has a mathematically exact threshold that can be deliberately shifted. Also, the bouncer can read context like "guest is in a bad mood" — the model only sees the features it receives.
σ(z) = 1 / (1 + e−z)
Formula: Sigmoid Function

This z is exactly the result of the weighted sum from the previous step — the bouncer’s internally calculated score. The sigmoid function squeezes this number into the range between 0 and 1. For large positive z it approaches 1, for large negative z it approaches 0, and at z = 0 it gives exactly 0.5. This turns an unbounded weighted sum into a probability.

Example: Spam Classifier

A spam classifier with three features: suspicious words, unknown sender, and urgency language.

Weighted sum z = 1.8. Sigmoid(1.8) ≈ 0.858.

Since 0.858 > 0.5, the model classifies the email as "Spam" with about 86% model-confidence. In a different use case, a hospital might lower the threshold to 0.3 to catch more at-risk patients — better to warn once too often than to miss a case.

Caution

Misconception: Logistic regression predicts numeric values like prices

The name comes from its statistical ancestry: logistic regression "regresses" on probabilities via the logit link. In practice, it is a classifier. Joseph Berkson coined the term "logit" in 1944 — the name "logistic regression" for such models only caught on later and has stuck despite the confusion.

Linear Regression

Output: continuous number (e.g., price in €). Use case: Regression (predicting values). Loss function: MSE.

Logistic Regression

Output: probability between 0 and 1. Use case: Classification (assigning classes). Loss function: Log Loss.

Interactive: The Sigmoid Formula Step by Step

Click on each term of the sigmoid formula to understand its role. The example shows a concrete spam classification with z = 2.5.

Sigmoid Formula

σ(z)1(1 + e⁻ᶻ)
σ(z) — Result

The output of the sigmoid function: a probability between 0 and 1. When σ(z) > 0.5, the model classifies as positive (e.g., spam); when σ(z) < 0.5, as negative.

Concrete Example

Given: Spam detection z = w·x + b = 0.5·5 + 0 = 2.5

Computation: z = w·x + b (Linear score) , σ(z) = 1/(1+e⁻ᶻ) (Sigmoid function)

1Compute exponential term: e^(−2.5) ≈ e^(−2.5) ≈ 0.082
2Compute denominator: 1 + 0.082 = 1 + 0.082 = 1.082
3Result: 1/1.082 ≈ 0.924 → Spam (> 0.5) 1/1.082 ≈ 0.924

The Big Picture — Bridge to the Artificial Neuron

What you learned in this article is more than three separate concepts. It is a chain that exactly matches the structure and training of an artificial neuron:

  1. 1. Weighted SumInputs × Weights + Bias → a number (just like linear regression)
  2. 2. Activation FunctionPass the number through a non-linear function (e.g., sigmoid)
  3. 3. Loss SignalCompare prediction to true value and measure the error (e.g., MSE)
  4. 4. Weight UpdateAdjust weights in the direction that shrinks the loss

You already understand 80% of an artificial neuron — you just don't know it yet. In the next learning path (Deep Learning), you will meet the neuron formally — and recognise that it is exactly this chain, just with more layers.

Adrien-Marie Legendre published the method of least squares in 1805 — the mathematical foundation on which MSE rests. Francis Galton made "regression" famous as an analysis tool in the 1880s: he observed that extremely tall parents tended to have children of more average height — a "regression to the mean." The name stuck, even though today it broadly refers to predicting numeric values from inputs. Joseph Berkson introduced the logit in 1944 and built on it a regression model for probabilities that today is mostly used as a classifier — a naming confusion that persists to this day.

Note

Key Takeaways

  • A linear model predicts by multiplying each input by a weight and adding a bias — the result is a straight line (or hyperplane).
  • MSE measures prediction quality by averaging squared errors, deliberately punishing large misses harder than small ones.
  • Logistic regression wraps the same weighted sum in a sigmoid so the output stays between 0 and 1 — turning a regression machine into a classifier.
  • Weights + bias + activation function + loss = the recipe card for almost every neural network.

Check

Comprehension Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: Linear & Logistic Regression

Not completed

In the equation ŷ = w · x + b — what does the term b represent?

Select one answer

Answer Key: 1) C · 2) B · 3) C · 4) B · 5) A · 6) C