Generative Image Models (Diffusion)
How an algorithm learned to generate dogs from snowfall — with the patience of a monk.
In the previous article, you learned how language models pick their next word. Now the same generative principle — start with randomness, apply learned structure — creates images.
This article reveals the surprisingly simple idea behind Stable Diffusion, Midjourney, and the now-discontinued DALL-E 2: teach a model to recognise noise, then run the process backwards. Not every image generator works this way: the image generation OpenAI introduced in ChatGPT in 2025 is, according to OpenAI, an autoregressive model that builds the image piece by piece, much like a language model builds its text.
Midjourney V5: Photorealistic AI Art
Photorealistic AI image generation reaches a new quality level, considerably transforming the creative industry. On March 15, 2023, Midjourney released Version 5 and achieved a quality leap that users described as 'uncanny' and 'too perfect.' The alpha version could for the first time generate photorealistic images that were barely distinguishable from real photographs. Particularly noteworthy: the chronic problem of malformed hands was considerably improved — V5 could correctly render five fingers in most cases. Julie Wieland, a graphic designer, compared the experience to 'finally getting glasses after ignoring poor vision for too long' — suddenly seeing everything in 4K quality [Source: Ars Technica, March 2023]. Improved prompt sensitivity enabled more precise creative control, while automatic upscaling scaled 1024x1024-pixel base images without additional GPU costs. V5 sparked intense debates about the future of human creativity.
Learning by Destruction — The Forward Process
Definition
Forward Process
Analogy
Definition
If the robot can do that, it can brush them off in reverse. Key insight: the robot learns grain patterns, not the painting itself — as a rule, the model does not memorise training images.
Imagine a 64x64 photo of a cat. Step 0: pristine photo. Step 250: clearly noisy, but the shape is still easy to see. Step 500: the shape can only just be made out. Step 750: practically nothing but noise. Step 1,000: pure random static, indistinguishable from any other noise image.
DDPM: Diffusion models established
The mathematical foundation of modern image generation through denoising processes. In June 2020, Jonathan Ho, Ajay Jain, and Pieter Abbeel published the influential paper 'Denoising Diffusion Probabilistic Models' – a class of latent variable models inspired by non-equilibrium thermodynamics. Their innovation lay in a weighted variational bound and the connection between diffusion models and denoising score matching with Langevin dynamics. The results were impressive: FID score of 3.17 on CIFAR-10 and Inception score of 9.46. DDPMs established a progressive lossy decompression approach that can be interpreted as a generalization of autoregressive decoding. This work laid the mathematical foundation for Stable Diffusion and the entire modern text-to-image generation.
Noise Steps
~1,000
Noise is added in ~1,000 small steps
Caution
Common Misconception: The Model Memorises Training Images
The model learns the statistical structure of noise removal, not specific images. However, with enough training on limited data, fragments of training images can be reproduced — which is why the copyright question is real and unresolved.
Why is noise added in a thousand tiny steps rather than all at once? Because each small step creates a learnable prediction task. One massive noise dump would leave the model with a nearly impossible task: recovering the whole image from pure noise in one go. The noise schedule (also called variance schedule) determines how much noise is added at each step — too little and the model learns trivial predictions, too much and structure is lost too early.
Creation from Chaos — The Reverse Process
Definition
Reverse Process
Analogy
Definition
There is no painting under the sand — the model creates structure from learned statistics, not from hidden data. Different sand piles (seeds) produce different paintings.
Stable Diffusion: start with random noise in latent space, run 50 denoising steps. Step 1: pure chaos. Step 10: vague shapes and colour patches. Step 25: recognisable object (cat outline visible). Step 50: sharp, detailed final image.
Denoising Steps
~50
In practice, ~50 steps suffice for image generation
Caution
Common Misconception: More Steps Always Produce Better Images
There are diminishing returns. Beyond roughly 50 steps, quality plateaus while compute cost keeps growing linearly. Modern schedulers like DDIM and DPM++ achieve good results in as few as 20 steps. Specially retrained (distilled) models even manage with 1 to 4 steps.
Making It Practical — Latent Diffusion & Text-Conditioning
Definition
Latent Diffusion
Analogy
Definition
The maquette captures essentials without knowing every detail of the original — just as the latent space compresses key image features. The client's brief guides but does not dictate — which is why unusual prompts can produce surprising results.
~786,000 values (512x512x3). Denoising directly on pixels. Requires data-centre GPUs with enormous memory. This is how early diffusion models such as DDPM (2020) worked, back then at no more than 256x256 pixels.
~16,000 values (64x64x4). Denoising on compressed representation. Runs on consumer GPUs. Stable Diffusion (Rombach et al., 2022).
Compression Factor
~50 x
~50x less data through latent-space compression
Caution
Common Misconception: The Model Understands My Prompt
The model follows statistical associations between text embeddings and image features learned during training. It has no concept of "dog" or "moon" — only learned correlations. This is why unusual prompts sometimes produce surprising or incoherent results.
The Complete Pipeline
Stable Diffusion Pipeline
- 1. Real imagetraining
- 2. Add noiseT steps
- 3. Pure noise
- 4. Predict & remove noiseT steps
- 5. Generated image
The forward process (training) and reverse process (generation) are mirror images: what is destroyed during training is reconstructed during generation.
Stable Diffusion: Image generation with open weights
The democratization of AI image generation through a powerful model with freely downloadable weights. On August 22, 2022, Stability AI released Stable Diffusion and significantly transformed access to advanced text-to-image technology. As a freely downloadable model (open weights under the CreativeML OpenRAIL-M licence, which forbids certain uses), Stable Diffusion could generate photorealistic 512x512-pixel images on consumer GPUs – an important advancement in speed and accessibility. Based on Latent Diffusion Models (LDMs), the system iterates through 'de-noising' in latent spaces instead of direct pixel manipulation. With 860 million parameters in the U-Net and 123 million in the text encoder, it remained relatively lightweight despite high performance. The GitHub-available source code enabled an explosively growing community to develop countless variants and tools. Stable Diffusion broke the monopoly of proprietary systems and made high-quality AI image generation accessible to everyone.
Interactive: How the Model Learns to Denoise
During training, the model sees an image that has been mixed with noise up to step t, and it has to predict exactly that noise. The box shows this with a cat head of 16 × 16 pixels: on the left the fading image, in the middle the noise, on the right the mix the model gets to see. The buttons jump through the steps of the example above, the slider through all 1,000.
Caution
Limitations & Copyright
- Diminishing returns: Beyond ~50 denoising steps, image quality barely improves while compute time grows linearly.
- Memorisation: In rare cases, diffusion models reproduce recognisable fragments of training images — especially when certain images appear disproportionately often in training.
- Copyright: Whether AI-generated images infringe the copyrights of training data is legally unresolved and the subject of active debate. First rulings exist, but none of them is final: in London, Getty Images largely lost its case against Stability AI in November 2025 but has been allowed to appeal. In the same month, the Munich Regional Court ruled that song lyrics memorised by the language model behind ChatGPT were reproductions; the case is now before the Higher Regional Court in Munich.
Note
Summary
- The forward process teaches the model what noise looks like at every stage of destruction — this is the training signal.
- The reverse process creates images by iteratively subtracting predicted noise from random static, step by step.
- Latent diffusion compresses images before denoising by ~50x, enabling generation on consumer GPUs.
- Text-conditioning via CLIP and cross-attention steers what emerges from the noise — but the model does not "understand" the prompt in any human sense.
This article completes Path I.G (Modern Architectures). From RNNs through attention, transformers, fine-tuning, and sampling, you've arrived at the most visible application of generative AI: image generation. Ready for the next learning path, Reinforcement Learning?
Check
Quiz: Generative Image Models (Diffusion)
What is the training objective of a diffusion model during the forward process?
Select one answer
1. What is the training objective of a diffusion model during the forward process?
2. A diffusion model with seed 42 generates an image of a cat. You change only the seed to 99 and keep everything else the same. What happens?
3. Pixel-space diffusion processes a 512x512x3 image (~786,000 values). Latent diffusion compresses this to 64x64x4 (~16,000 values). By roughly what factor is the data reduced?
4. A user prompts "a red car on a beach" but the generated image shows a blue car on grass. Which part of the pipeline most likely caused this?
5. After 50 denoising steps the image looks sharp. A user runs 200 steps instead. What is the most likely outcome?
6. Why is latent diffusion considered a breakthrough compared to earlier pixel-space diffusion models?
Check
Knowledge Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done