The Raw Material: Data Engineering for Machine Learning

You've learned how models optimize their parameters via gradient descent and regression. But what goes INTO the model? Raw data from the real world is messy, incomplete, and measured on wildly different scales.

This article covers the three pillars of data preparation — the work that happens before a single training step. Because the most expensive GPU cluster in the world can't save a model trained on garbage data.

Feature Engineering: Choosing the Right Inputs

Think of Feature Engineering as preparing ingredients for a recipe. You can have the best kitchen equipment and the most skilled chef, but if the ingredients are rotten, poorly cut, or missing key flavors, the dish will fail. Selecting the right ingredients (feature selection), preparing them correctly (feature creation), and measuring them precisely (encoding) determines the outcome before cooking even begins.

Example

In cooking, a skilled chef can sometimes rescue poor ingredients through technique. An ML algorithm cannot — it is purely mechanical and has no intuition to compensate for bad features.

Analogy:

Think of Feature Engineering as preparing ingredients for a recipe. You can have the best kitchen equipment and the most skilled chef, but if the ingredients are rotten, poorly cut, or missing key flavors, the dish will fail. Selecting the right ingredients (feature selection), preparing them correctly (feature creation), and measuring them precisely (encoding) determines the outcome before cooking even begins.

Example

In cooking, a skilled chef can sometimes rescue poor ingredients through technique. An ML algorithm cannot — it is purely mechanical and has no intuition to compensate for bad features.

Definition:

Feature Engineering is the process of selecting, transforming, and creating input variables (features) from raw data so that a machine learning algorithm can effectively learn patterns. Core principle: the algorithm can only find what the data makes visible. Garbage In, Garbage Out.

Consider a concrete example: real estate price prediction. The raw data contains an address as text — practically useless to a model. Feature Engineering transforms it into meaningful numbers:

1

Raw Data Address: "Musterstr. 42, 80331 Munich" — plain text, not usable

2

Extraction Extract postal code → 80331, map to district → "Altstadt-Lehel"

3

Engineered Feature Look up median household income for district → €58,200/year — highly predictive!

Categorical data like colors (red, blue, green) requires special treatment: One-Hot Encoding creates a separate binary column for each value (is_red, is_blue, is_green). This lets the algorithm work with categorical data without imposing an artificial ordering.

Wrong. The Curse of Dimensionality shows: beyond a certain number of features, models get WORSE, not better. The data space becomes so sparsely populated that the algorithm can no longer find reliable patterns. Better to have 5 informative features than 50 irrelevant ones.

Feature quality outweighs model complexity. A simple linear model with excellent features regularly beats a complex neural network with bad features.

Interactive: The Data Engineering Pipeline

Press Play and watch how raw data gets prepared step by step.

Data Pipeline Visualized

The typical data preparation process in five steps: From messy raw data through cleaning, feature engineering, and scaling to a model-ready dataset.

Step 0 of 5

Start animation

Click "Play" to see the RAG pipeline step by step.

Core Principle

The pipeline is strictly sequential: each step builds on the previous one. The order is critical — scaling before the split causes data leakage.

Missing Values: Holes in the Data

Imagine a patient chart with some fields left blank. Do you (a) throw away the entire patient record (deletion), (b) fill in the average value from all other patients (imputation), or (c) add a note "this field was left blank" — which itself might be diagnostically relevant (indicator variable)? Each choice has consequences.

Example

In medicine, a doctor uses clinical judgment to decide. In ML, the choice is mechanical and must be made before training, not during.

Analogy:

Imagine a patient chart with some fields left blank. Do you (a) throw away the entire patient record (deletion), (b) fill in the average value from all other patients (imputation), or (c) add a note "this field was left blank" — which itself might be diagnostically relevant (indicator variable)? Each choice has consequences.

Example

In medicine, a doctor uses clinical judgment to decide. In ML, the choice is mechanical and must be made before training, not during.

Definition:

Missing values are data points where one or more features have no recorded value. They arise from sensor failures, incomplete forms, data transfer errors, or intentional non-responses. Most ML algorithms crash or produce meaningless results when encountering empty fields.

There are three established strategies, each with clear pros and cons:

Deletion (Remove)

Simple but wasteful: Remove all rows with missing values. With 20% missing values, the dataset shrinks from 1,000 to 800 entries.

Imputation (Replace)

Preserves data but introduces assumptions: Fill empty fields with median or mean. Keeps all 1,000 entries, but assumes "average health."

The cleverest option: Create an additional column "bp_missing=1". The model might discover that missing blood pressure correlates with emergency admissions — patients too critical for measurement. The absence of data becomes an informative signal itself.

Concrete example: A medical dataset has 20% missing blood pressure values. You suspect the data is missing because obese patients refused to be measured. Deletion loses 200 records. Median imputation with 120 mmHg assumes "normal value." An indicator variable reveals the actual reason.

~80%

of data science work time goes into data preparation

Normalization & Standardization: Getting Everything on the Same Scale

Imagine comparing the distance from Berlin to Munich with the number of rooms in an apartment. One is measured in kilometers (hundreds), the other in single digits. If you plotted both on the same axis, the kilometer values would completely dwarf the room count. Scaling is like unit conversion: the actual distances and relationships between data points stay identical, only the measuring scale changes.

Example

In the real world, you simply choose appropriate units. In ML, scaling is a mathematical transformation that must be computed from training data and applied consistently to test data.

Analogy:

Imagine comparing the distance from Berlin to Munich with the number of rooms in an apartment. One is measured in kilometers (hundreds), the other in single digits. If you plotted both on the same axis, the kilometer values would completely dwarf the room count. Scaling is like unit conversion: the actual distances and relationships between data points stay identical, only the measuring scale changes.

Example

In the real world, you simply choose appropriate units. In ML, scaling is a mathematical transformation that must be computed from training data and applied consistently to test data.

Definition:

Feature scaling transforms all features to comparable ranges so that no single feature dominates the learning process due to its numeric scale. Two main methods: Min-Max Normalization (scales to [0,1]) and Z-Standardization (scales to mean=0, std=1).

Min-Max Normalization

x_norm = (x - x_min) / (x_max - x_min)

Z-Standardization (Z-Score)

z = (x - μ) / σ

Three features with completely different ranges: Age (0–100), Income (0–1,000,000), Rooms (1–10). Data point: Age=50, Income=60,000, Rooms=4.

Min-Max Normalization:
Age:    (50 - 0) / (100 - 0)           = 0.50
Income: (60,000 - 0) / (1,000,000 - 0) = 0.06
Rooms:  (4 - 1) / (10 - 1)             = 0.33

All values now in [0, 1].

Z-Standardization (given mean/std):
Age:    (50 - 40) / 15       = 0.67
Income: (60,000 - 50,000) / 25,000 = 0.40
Rooms:  (4 - 3) / 1.5        = 0.67

All values now comparable.

Without scaling, the gradient descent update for Income would be ~10,000x larger than for Rooms — causing zigzag convergence instead of a direct path to the minimum.

Unscaled features create elongated contours in the loss landscape. Gradient descent then zigzags across the narrow valley instead of heading straight to the minimum. Scaled features create approximately circular contours — the gradient points directly to the minimum. Result: significantly fewer iterations until convergence.

No. Normalization changes the unit of measurement, not the relationships between data points. After Min-Max, the formerly largest value is still the largest (now 1.0) and the smallest is still the smallest (now 0.0). The data structure remains intact.

Data leakage occurs when information from test data accidentally contaminates training. Most common mistake: fitting the scaler (e.g., StandardScaler in scikit-learn) on the ENTIRE dataset (train + test) before splitting. Then the scaler already knows test data statistics — and the model appears better than it actually is. You don't need to be able to write the following code — just pay attention to the logical difference in what the scaler is applied to:

# WRONG: Leakage!
scaler.fit(entire_dataset)
X_train, X_test = split(entire_dataset)

# CORRECT: No leakage
X_train, X_test = split(entire_dataset)
scaler.fit(X_train)          # Train only!
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

Interactive: Sort Pipeline Steps

In what order must data preparation steps be executed? Sort the pipeline — but beware: the wrong order causes data leakage!

Sort the pipeline steps into the correct order. Hint: Think about data leakage — which step absolutely must come early?

First Step

1.Scale features

2.Train/Test Split

3.Handle missing values

4.Feature Engineering

5.Load raw data

Last Step

A model is only as good as its features — Feature Engineering (selecting, creating, and encoding the right inputs) is the single biggest lever for model quality.
Real data has holes. Deletion is simple but wasteful, imputation preserves data but introduces assumptions, and indicator variables can turn the absence of data into a useful signal.
Features on different scales sabotage gradient descent. Normalization (0–1) and standardization (mean=0, std=1) fix this — they change the unit of measurement without changing the data's meaning.

Learning Goals

Explain with your own example why raw text data (e.g., an address) is not a good feature for an ML model.
You have a dataset with 30% missing temperature values from a sensor. Which of the three strategies would you choose and why?
Calculate the Min-Max normalization for x=75 with a range of 0 to 200.

Your dataset has a "Color" column with values red, blue, green. What must you do before feeding it into a linear regression model?

Delete the column — colors aren't numeric.

Replace each color with a number: red=1, blue=2, green=3.

Apply One-Hot Encoding: create three binary columns (is_red, is_blue, is_green).

Normalize the color values to [0,1].

1. Your dataset has a "Color" column with values red, blue, green. What must you do before feeding it into a linear regression model?

☐ A) Delete the column — colors aren't numeric.
☐ B) Replace each color with a number: red=1, blue=2, green=3.
☐ C) Apply One-Hot Encoding: create three binary columns (is_red, is_blue, is_green).
☐ D) Normalize the color values to [0,1].

2. A medical dataset has 15% missing weight values. You suspect the data is missing because obese patients refused to be weighed. Which strategy best preserves this information?

☐ A) Delete all rows with missing weight.
☐ B) Fill missing values with the column median.
☐ C) Create an indicator variable "weight_missing" alongside median imputation.
☐ D) Fill missing values with 0.

3. Feature A ranges from 0–1, Feature B from 0–1,000,000. You train a model with gradient descent WITHOUT scaling. What happens?

☐ A) The model ignores Feature A entirely.
☐ B) Gradient updates are dominated by Feature B, causing slow zigzag convergence.
☐ C) The model automatically adjusts for different scales.
☐ D) The loss function becomes convex.

4. You scale your data using StandardScaler fitted on the ENTIRE dataset (train + test), then split and evaluate. Your test accuracy is 95%. A colleague points out a methodological flaw. What is it?

☐ A) StandardScaler should use Min-Max instead.
☐ B) You introduced data leakage — the scaler learned statistics from test data, inflating test performance.
☐ C) 95% accuracy is too high, so the model must be overfitting.
☐ D) Scaling should only be applied to categorical features.

Answer Key: 1) C · 2) C · 3) B · 4) B

The Raw Material: Data Engineering for Machine Learning

Feature Engineering: Choosing the Right Inputs

Feature Engineering

Example

Analogy:

Example

Definition:

Misconception: More Features = Better Model

Interactive: The Data Engineering Pipeline

Missing Values: Holes in the Data

Missing Values

Example

Analogy:

Example

Definition:

Third Strategy: Indicator Variable

Normalization & Standardization: Getting Everything on the Same Scale

Feature Scaling

Example

Analogy:

Example

Definition:

Worked Example: Scaling Three Features

Deep Dive: Why Scaling Speeds Up Gradient Descent

Misconception: Normalization Changes the Data

Deep Dive: Data Leakage — The Hidden Trap

Interactive: Sort Pipeline Steps

Key Takeaways

Learning Goals

Test Your Knowledge

Your dataset has a "Color" column with values red, blue, green. What must you do before feeding it into a linear regression model?

Feature Engineering: Choosing the Right Inputs

Feature Engineering

Example

Analogy:

Example

Definition:

Misconception: More Features = Better Model

Interactive: The Data Engineering Pipeline

Missing Values: Holes in the Data

Missing Values

Example

Analogy:

Example

Definition:

Third Strategy: Indicator Variable

Normalization & Standardization: Getting Everything on the Same Scale

Feature Scaling

Example

Analogy:

Example

Definition:

Worked Example: Scaling Three Features

Deep Dive: Why Scaling Speeds Up Gradient Descent

Misconception: Normalization Changes the Data

Deep Dive: Data Leakage — The Hidden Trap

Interactive: Sort Pipeline Steps

Key Takeaways

Learning Goals

Test Your Knowledge

Your dataset has a "Color" column with values red, blue, green. What must you do before feeding it into a linear regression model?

Related Content

Article

Supervised Learning — Learning with a Teacher

When the Model Memorizes (Overfitting)

The Machine's Knobs — Parameters vs. Hyperparameters

Measures of Central Tendency: Where Is the Middle?

Data Structures III (Key-Based)

Functions

Version Control with Git

The Path to the Valley: Gradient Descent

Linear & Logistic Regression

Lists and Dictionaries

Programming vs. Training

How Good Is Your Model? Metrics That Actually Matter

External Code Libraries (Packages)

Connecting Your Own Data (RAG)

Unsupervised Learning

Demo

K-Means Clustering

Glossary

Timeline