Back to Learning Area

Generative Image Models (Diffusion)

How an algorithm learned to generate dogs from snowfall — with the patience of a monk.

  • Architectures
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Creating images from noise

What is a diffusion model?

A diffusion model is an AI program. It creates images. You enter text. The program creates a matching image. Famous programs are DALL-E and Stable Diffusion.

How images are born

The model learns from many images. During training, you add noise. Noise means: random dots without meaning. The model learns: what did the image look like before? Finally, it can remove the noise. This is how an image emerges from noise.

Stable Diffusion on a PC

Early image models needed data centers. Stable Diffusion is a free program. It runs on a normal home PC. It uses a latent space. Latent space means: the program shrinks the image first. This saves computing power.

In the previous article, you learned how language models pick their next word. Now the same generative principle — start with randomness, apply learned structure — creates images.

This article reveals the surprisingly simple idea behind Stable Diffusion, Midjourney, and the now-discontinued DALL-E 2: teach a model to recognise noise, then run the process backwards. Not every image generator works this way: the image generation OpenAI introduced in ChatGPT in 2025 is, according to OpenAI, an autoregressive model that builds the image piece by piece, much like a language model builds its text.

Learning by Destruction — The Forward Process

Definition

Forward Process

Analogy

Imagine a painting on a table. You throw a thin layer of sand on it, then another, then another — a thousand times. By the end, the painting is invisible under sand. Training a diffusion model is like training a robot to recognise, at every single layer, exactly which grains were just added.

If the robot can do that, it can brush them off in reverse. Key insight: the robot learns grain patterns, not the painting itself — as a rule, the model does not memorise training images.

Imagine a 64x64 photo of a cat. Step 0: pristine photo. Step 250: clearly noisy, but the shape is still easy to see. Step 500: the shape can only just be made out. Step 750: practically nothing but noise. Step 1,000: pure random static, indistinguishable from any other noise image.

Noise Steps

~1,000

Noise is added in ~1,000 small steps

Caution

Common Misconception: The Model Memorises Training Images

The model learns the statistical structure of noise removal, not specific images. However, with enough training on limited data, fragments of training images can be reproduced — which is why the copyright question is real and unresolved.

Why is noise added in a thousand tiny steps rather than all at once? Because each small step creates a learnable prediction task. One massive noise dump would leave the model with a nearly impossible task: recovering the whole image from pure noise in one go. The noise schedule (also called variance schedule) determines how much noise is added at each step — too little and the model learns trivial predictions, too much and structure is lost too early.

Creation from Chaos — The Reverse Process

Definition

Reverse Process

Analogy

The trained robot from Concept 1 now works on a fresh, random pile of sand (no painting underneath). Using everything it learned about grain patterns, it brushes away one layer at a time. Because the robot's expertise encodes the structure of "real paintings," the sand naturally settles into a painting-like pattern.

There is no painting under the sand — the model creates structure from learned statistics, not from hidden data. Different sand piles (seeds) produce different paintings.

Stable Diffusion: start with random noise in latent space, run 50 denoising steps. Step 1: pure chaos. Step 10: vague shapes and colour patches. Step 25: recognisable object (cat outline visible). Step 50: sharp, detailed final image.

Denoising Steps

~50

In practice, ~50 steps suffice for image generation

Caution

Common Misconception: More Steps Always Produce Better Images

There are diminishing returns. Beyond roughly 50 steps, quality plateaus while compute cost keeps growing linearly. Modern schedulers like DDIM and DPM++ achieve good results in as few as 20 steps. Specially retrained (distilled) models even manage with 1 to 4 steps.

Making It Practical — Latent Diffusion & Text-Conditioning

Definition

Latent Diffusion

Analogy

Instead of sculpting a life-size marble statue directly (pixel space), you first sculpt a small clay maquette (latent space). The maquette captures all essential proportions in miniature. Only when it's finished does a machine scale it up to full size (VAE decoder). Meanwhile, a client's brief (text prompt via CLIP) tells the sculptor what to carve at every step.

The maquette captures essentials without knowing every detail of the original — just as the latent space compresses key image features. The client's brief guides but does not dictate — which is why unusual prompts can produce surprising results.

Pixel Space

~786,000 values (512x512x3). Denoising directly on pixels. Requires data-centre GPUs with enormous memory. This is how early diffusion models such as DDPM (2020) worked, back then at no more than 256x256 pixels.

Latent Space

~16,000 values (64x64x4). Denoising on compressed representation. Runs on consumer GPUs. Stable Diffusion (Rombach et al., 2022).

Compression Factor

~50 x

~50x less data through latent-space compression

Caution

Common Misconception: The Model Understands My Prompt

The model follows statistical associations between text embeddings and image features learned during training. It has no concept of "dog" or "moon" — only learned correlations. This is why unusual prompts sometimes produce surprising or incoherent results.

The Complete Pipeline

Stable Diffusion Pipeline

Enter text prompt
CLIP encodes the text into embedding vectors (one per token)
Generate random noise in latent space (64x64x4)
U-Net denoises in ~50 steps (cross-attention with text embedding)
VAE decoder converts latent image to 512x512 pixel image
  1. 1. Real imagetraining
  2. 2. Add noiseT steps
  3. 3. Pure noise
  4. 4. Predict & remove noiseT steps
  5. 5. Generated image

The forward process (training) and reverse process (generation) are mirror images: what is destroyed during training is reconstructed during generation.

Interactive: How the Model Learns to Denoise

During training, the model sees an image that has been mixed with noise up to step t, and it has to predict exactly that noise. The box shows this with a cat head of 16 × 16 pixels: on the left the fading image, in the middle the noise, on the right the mix the model gets to see. The buttons jump through the steps of the example above, the slider through all 1,000.

Caution

Limitations & Copyright

  • Diminishing returns: Beyond ~50 denoising steps, image quality barely improves while compute time grows linearly.
  • Memorisation: In rare cases, diffusion models reproduce recognisable fragments of training images — especially when certain images appear disproportionately often in training.
  • Copyright: Whether AI-generated images infringe the copyrights of training data is legally unresolved and the subject of active debate. First rulings exist, but none of them is final: in London, Getty Images largely lost its case against Stability AI in November 2025 but has been allowed to appeal. In the same month, the Munich Regional Court ruled that song lyrics memorised by the language model behind ChatGPT were reproductions; the case is now before the Higher Regional Court in Munich.

Note

Summary

  1. The forward process teaches the model what noise looks like at every stage of destruction — this is the training signal.
  2. The reverse process creates images by iteratively subtracting predicted noise from random static, step by step.
  3. Latent diffusion compresses images before denoising by ~50x, enabling generation on consumer GPUs.
  4. Text-conditioning via CLIP and cross-attention steers what emerges from the noise — but the model does not "understand" the prompt in any human sense.

This article completes Path I.G (Modern Architectures). From RNNs through attention, transformers, fine-tuning, and sampling, you've arrived at the most visible application of generative AI: image generation. Ready for the next learning path, Reinforcement Learning?

Check

Quiz: Generative Image Models (Diffusion)

Not completed

What is the training objective of a diffusion model during the forward process?

Select one answer

Answer Key: 1) B · 2) B · 3) C · 4) C · 5) B · 6) C

Check

Knowledge Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done