How an algorithm learned to generate dogs from snowfall — with the patience of a monk.
Architectures 10 min Intermediate June 15, 2026
In the previous article, you learned how language models pick their next word. Now the same generative principle — start with randomness, apply learned structure — creates images.
This article reveals the surprisingly simple idea behind DALL-E, Midjourney, and Stable Diffusion: teach a model to recognise noise, then run the process backwards.
2021 Products
DALL-E Creates Images from Text
A landmark breakthrough in text-to-image generation and a meaningful advance in AI creativity. On January 5, 2021, OpenAI unveiled DALL-E — a system that generates coherent and often remarkably creative images from text descriptions. Text-to-image models had existed before (such as alignDRAW in 2015 or GAN-based approaches like StackGAN and AttnGAN), but DALL-E raised coherence and versatility to a new level. Based on a 12-billion-parameter version of GPT-3, DALL-E demonstrated that the boundary between language and visual understanding can be crossed. The system trained on 250 million image-text pairs from the internet and developed noteworthy capabilities in the process: it can anthropomorphize animals, plausibly combine unrelated concepts, and even render text within images. Mark Riedl of Georgia Tech commented that the results were 'remarkably more coherent' than any previous text-to-image system. DALL-E successfully extended GPT's language understanding into the visual domain and opened an entirely new dimension of AI creativity.
Learning by Destruction — The Forward Process
Forward Process
AnalogyDefinition
Imagine a painting on a table. You throw a thin layer of sand on it, then another, then another — a thousand times. By the end, the painting is invisible under sand. Training a diffusion model is like training a robot to recognise, at every single layer, exactly which grains were just added.
Analogy:
Imagine a painting on a table. You throw a thin layer of sand on it, then another, then another — a thousand times. By the end, the painting is invisible under sand. Training a diffusion model is like training a robot to recognise, at every single layer, exactly which grains were just added.
Definition:
The forward process takes a real training image and adds small, precisely calibrated amounts of Gaussian noise over T steps (typically ~1,000). By step T, the image is pure static. The model's training task: predict which specific noise was added at each step.
If the robot can do that, it can brush them off in reverse. Key insight: the robot learns grain patterns, not the painting itself — the model does not memorise training images.
Imagine a 64x64 photo of a cat. Step 0: pristine photo. Step 250: slightly fuzzy, details blurred. Step 500: recognisable shape but colours washed out. Step 750: vague blob. Step 1,000: pure random static, indistinguishable from any other noise image.
2020 Papers
DDPM: Diffusion models established
The mathematical foundation of modern image generation through denoising processes. In June 2020, Jonathan Ho, Ajay Jain, and Pieter Abbeel published the influential paper 'Denoising Diffusion Probabilistic Models' – a class of latent variable models inspired by non-equilibrium thermodynamics. Their innovation lay in a weighted variational bound and the connection between diffusion models and denoising score matching with Langevin dynamics. The results were impressive: FID score of 3.17 on CIFAR-10 and Inception score of 9.46. DDPMs established a progressive lossy decompression approach that can be interpreted as a generalization of autoregressive decoding. This work laid the mathematical foundation for Stable Diffusion and the entire modern text-to-image generation.
~1,000
Training Steps Noise is added in ~1,000 small steps
Common Misconception: The Model Memorises Training Images
The model learns the statistical structure of noise removal, not specific images. However, with enough training on limited data, fragments of training images can be reproduced — which is why the copyright question is real and unresolved.
The Noise Schedule
Why is noise added in a thousand tiny steps rather than all at once? Because each small step creates a learnable prediction task. One massive noise dump gives the model no gradient signal to learn from. The variance schedule determines how much noise is added at each step — too little and the model learns trivial predictions, too much and structure is lost too early.
Creation from Chaos — The Reverse Process
Reverse Process
AnalogyDefinition
The trained robot from Concept 1 now works on a fresh, random pile of sand (no painting underneath). Using everything it learned about grain patterns, it brushes away one layer at a time. Because the robot's expertise encodes the structure of "real paintings," the sand naturally settles into a painting-like pattern.
Analogy:
The trained robot from Concept 1 now works on a fresh, random pile of sand (no painting underneath). Using everything it learned about grain patterns, it brushes away one layer at a time. Because the robot's expertise encodes the structure of "real paintings," the sand naturally settles into a painting-like pattern.
Definition:
The reverse process starts from pure random noise and runs the trained model backwards through T steps. At each step, the model predicts the noise present in the current image and subtracts it. After all steps, a coherent image emerges. The seed (random starting point) determines which specific image appears.
There is no painting under the sand — the model creates structure from learned statistics, not from hidden data. Different sand piles (seeds) produce different paintings.
Stable Diffusion: start with random noise in latent space, run 50 denoising steps. Step 1: pure chaos. Step 10: vague shapes and colour patches. Step 25: recognisable object (cat outline visible). Step 50: sharp, detailed final image.
~50
Denoising Steps In practice, ~50 steps suffice for image generation
Common Misconception: More Steps Always Produce Better Images
There are diminishing returns. Beyond roughly 50 steps, quality plateaus while compute cost keeps growing linearly. Modern schedulers like DDIM and DPM++ achieve good results in as few as 20 steps.
Making It Practical — Latent Diffusion & Text-Conditioning
Latent Diffusion
AnalogyDefinition
Instead of sculpting a life-size marble statue directly (pixel space), you first sculpt a small clay maquette (latent space). The maquette captures all essential proportions in miniature. Only when it's finished does a machine scale it up to full size (VAE decoder). Meanwhile, a client's brief (text prompt via CLIP) tells the sculptor what to carve at every step.
Analogy:
Instead of sculpting a life-size marble statue directly (pixel space), you first sculpt a small clay maquette (latent space). The maquette captures all essential proportions in miniature. Only when it's finished does a machine scale it up to full size (VAE decoder). Meanwhile, a client's brief (text prompt via CLIP) tells the sculptor what to carve at every step.
Definition:
Latent diffusion runs the denoising process not on raw pixels but on a compressed representation (latent space) produced by a specialised compression network (VAE encoder). This reduces data from ~786,000 values (512x512x3) to ~16,000 (64x64x4) — a ~50x compression. After denoising, the VAE decoder reconstructs the full-resolution pixel image. Text-conditioning via CLIP: the text prompt is converted to a numerical representation, and a specialised denoising network uses it like a steering wheel — each denoising step is guided toward the prompt.
The maquette captures essentials without knowing every detail of the original — just as the latent space compresses key image features. The client's brief guides but does not dictate — which is why unusual prompts can produce surprising results.
Pixel Space
~786,000 values (512x512x3). Denoising directly on pixels. Requires data-centre GPUs with enormous memory. Historically first diffusion models (DDPM, 2020).
Latent Space
~16,000 values (64x64x4). Denoising on compressed representation. Runs on consumer GPUs. Stable Diffusion (Rombach et al., 2022).
~50x
Compression Factor ~50x less data through latent-space compression
Common Misconception: The Model Understands My Prompt
The model follows statistical associations between text embeddings and image features learned during training. It has no concept of "dog" or "moon" — only learned correlations. This is why unusual prompts sometimes produce surprising or incoherent results.
The Complete Pipeline
Stable Diffusion Pipeline
Enter text prompt
CLIP encodes the text into an embedding vector
Generate random noise in latent space (64x64x4)
U-Net denoises in ~50 steps (cross-attention with text embedding)
VAE decoder converts latent image to 512x512 pixel image
1
Real image (training)
2
Add noise (T steps)
3
Pure noise
4
Predict & remove noise (T steps)
5
Generated image
The forward process (training) and reverse process (generation) are mirror images: what is destroyed during training is reconstructed during generation.
2022 Products
Stable Diffusion: Open-source image generation
The democratization of AI image generation through the first powerful open-source model. On August 22, 2022, Stability AI released Stable Diffusion and significantly transformed access to advanced text-to-image technology. As the first open-source model of its class, Stable Diffusion could generate photorealistic 512x512-pixel images on consumer GPUs – an important advancement in speed and accessibility. Based on Latent Diffusion Models (LDMs), the system iterates through 'de-noising' in latent spaces instead of direct pixel manipulation. With 860 million parameters in the U-Net and 123 million in the text encoder, it remained relatively lightweight despite high performance. The GitHub-available source code enabled an explosively growing community to develop countless variants and tools. Stable Diffusion broke the monopoly of proprietary systems and made high-quality AI image generation accessible to everyone.
Interactive: How the Model Learns to Denoise
During training, the U-Net learns to predict the noise that was added. In each step, it compares its prediction with the actual noise and adjusts its weights. Watch how the prediction error (loss) decreases with each optimization step — this exact process is what makes the model increasingly better at reconstructing images from noise.
1
2
3
4
5
Step 1 / 5Starting position
The ball starts far left of the minimum. The loss is high (L=105.8). The gradient points steeply downhill to the right — the algorithm knows which direction to go.
Limitations & Copyright
Diminishing returns: Beyond ~50 denoising steps, image quality barely improves while compute time grows linearly.
Memorisation: In rare cases, diffusion models reproduce recognisable fragments of training images — especially when certain images appear disproportionately often in training.
Copyright: Whether AI-generated images infringe the copyrights of training data is legally unresolved and the subject of active debate.
Summary
The forward process teaches the model what noise looks like at every stage of destruction — this is the training signal.
The reverse process creates images by iteratively subtracting predicted noise from random static, step by step.
Latent diffusion compresses images before denoising by ~50x, enabling generation on consumer GPUs.
Text-conditioning via CLIP and cross-attention steers what emerges from the noise — but the model does not "understand" the prompt in any human sense.
This article completes Path I.G (Modern Architectures). From RNNs through attention, transformers, fine-tuning, and sampling, you've arrived at the most visible application of generative AI: image generation. Ready for the next sector?
Quiz: Generative Image Models (Diffusion)
Question 1 / 6
Not completed
What is the training objective of a diffusion model during the forward process?
1. What is the training objective of a diffusion model during the forward process?
☐ A) Memorise each training image
☐ B) Predict which noise was added at each step
☐ C) Compress images into latent space
☐ D) Generate new images from text
2. A diffusion model with seed 42 generates an image of a cat. You change only the seed to 99 and keep everything else the same. What happens?
☐ A) The same cat appears
☐ B) A completely different image appears
☐ C) The image quality improves
☐ D) The model refuses to generate
3. Pixel-space diffusion processes a 512x512x3 image (~786,000 values). Latent diffusion compresses this to 64x64x4 (~16,000 values). By roughly what factor is the data reduced?
☐ A) ~4x
☐ B) ~16x
☐ C) ~50x
☐ D) ~100x
4. A user prompts "a red car on a beach" but the generated image shows a blue car on grass. Which part of the pipeline most likely caused this?
☐ A) The VAE decoder distorted the colours
☐ B) The seed was unlucky
☐ C) The CLIP text encoder produced an embedding that poorly captured the prompt's meaning
☐ D) The forward process destroyed too much information
5. After 50 denoising steps the image looks sharp. A user runs 200 steps instead. What is the most likely outcome?
☐ A) A dramatically better image
☐ B) A nearly identical image but with 4x the compute time
☐ C) A worse image due to over-processing
☐ D) The model crashes
6. Why is latent diffusion considered a breakthrough compared to earlier pixel-space diffusion models?
☐ A) It produces higher-resolution images
☐ B) It eliminates the need for text prompts
☐ C) It compresses images before denoising, enabling generation on consumer hardware
☐ D) It uses a different type of noise
Answer Key: 1) B · 2) B · 3) C · 4) C · 5) B · 6) C
Knowledge Check
Why do developers train the model to predict noise rather than simply memorising the finished image?
How does a finished image emerge from pure random noise, and why does a different seed produce a completely different result?
Why is it possible to run a modern image model like Stable Diffusion on a decent home computer, even though the first models required huge servers?