Back to Learning Area

Quantization — Shrinking Models

Shrinking models for home use — the technique that makes local AI practical in the first place.

  • Concepts
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Shrinking large AI models

What is quantization?

An AI model consists of billions of numbers. These numbers are called weights. Quantization makes the numbers less accurate. This makes the model smaller. For example: FP16 uses 16 bits per number. INT4 uses only 4 bits. The model shrinks by 70 percent. The quality barely decreases.

GGUF: one file for everything

GGUF is a file format for local AI models. A GGUF file contains everything. You do not have to assemble anything. Q4_K_M is the recommended variant for most users.

Does the model fit on my PC?

The model must fit into the graphics card memory. This memory is called VRAM. With 8 GB VRAM, a 7B-Q4_K_M model fits well. A larger model with less accuracy often beats a small model with high accuracy.

The latest open-source language models are incredibly capable — but they weigh 14 GB or more. Your gaming laptop might have 8 GB of VRAM. Does that mean local AI is out of reach?

Not at all. Quantization compresses these models by reducing the precision of their billions of weights — shrinking them by up to 80% while keeping most of their intelligence intact. This article shows you exactly how it works and which model fits your machine.

Precision — How Many Bits Per Number?

Definition

Quantization

Analogy

Think of the number Pi. At full precision (FP32), you store "3.1415927" — perfect for calculations. At half precision (FP16), you store "3.141" — good enough for almost everything. At 4-bit (INT4), you store just "3" — a rough approximation, but still correct enough to know that a circle with diameter 1 has a circumference of about 3.

Unlike rounding a single number, quantizing a neural network affects billions of weights simultaneously. The interactions between rounded weights can compound errors unpredictably. However, in practice, modern quantization methods distribute errors so well that quality loss stays minimal.

FP32 (Full Precision) 32 bits = 4 bytes per weight. 7B model: ~28 GB
FP16 (Half Precision) 16 bits = 2 bytes per weight. 7B model: ~14 GB
INT8 8 bits = 1 byte per weight. 7B model: ~7 GB
INT4 4 bits = 0.5 bytes per weight. 7B model: ~3.5 GB

Llama 3.1 8B: FP16 = 16 GB, Q8_0 = 8.5 GB, Q4_K_M = 4.9 GB, Q2_K = 3.2 GB. Why 4.9 GB and not the calculated 4 GB? Q4_K_M uses about 4.9 bits per weight on average, because scaling factors and a few weight matrices kept at higher precision are stored as well. From FP16 to Q4_K_M, the model shrinks by ~70%. Measured quality loss on benchmarks: under 5% for most tasks.

Size Reduction

~70 %

~70% smaller from FP16 to Q4_K_M

Quality Loss

<5 %

Under 5% for Q4_K_M on most tasks

Caution

Common Misconception: Quantization Destroys Model Quality

At 4 bits (Q4_K_M), quality loss is typically under 5% on standard benchmarks. For Q8_0, it is nearly indistinguishable from FP16. Below Q4 the loss rises quickly: at Q3 it becomes noticeable, at Q2 substantial — but Q4 and above preserve the vast majority of model capability.

Interactive: What Does Each Precision Level Cost?

The article explained the four precision levels FP32, FP16, INT8, and INT4. Here you see them as an interactive comparison matrix: click a row and observe how size, speed, and quality behave at each level. The heatmap colors instantly show you where the advantages and trade-offs lie.

Precision Matrix: Quantization Levels Compared

Advantage (small/fast/high)
Neutral
Disadvantage (large/slow/low)

Click a row for detail view

FormatBits/ParamModel SizeInference SpeedQualityVRAM (7B)
FP3232 100 % 1x 100 % ~28 GB
FP1616 50 % 2x 99 % ~14 GB
INT88 25 % 3x 99 % ~7 GB
INT44 12.5 % 5x 96 % ~3.5 GB
Key Takeaway: Quantization is a trade-off. INT4 models need only 12.5% of the memory and run up to 5x faster — with usually less than 5% quality loss. For many applications, that's an excellent deal.

GGUF — One File, Everything Included

GGUF is like a self-contained app installer — as opposed to the old approach of manually downloading separate parts (weights, config, tokenizer) and assembling them yourself. One file, double-click, done.

GGUF (the successor to the older GGML format) is the standard file format for running local language models, developed for llama.cpp. It is monolithic: one file contains model weights, tokenizer, configuration, and metadata. The naming convention encodes everything: model-params-quant.gguf. Q-levels (Q2_K through Q8_0) indicate the quantization level. The K suffix means K-Quant (block-wise quantization, partly with mixed precision), S/M/L indicates how many weights are stored at higher precision. llama.cpp also knows further levels, such as the IQ variants and 4-bit floating-point formats like MXFP4 (used by gpt-oss) and, since March 2026, NVIDIA’s NVFP4.

Unlike an app installer which deletes itself after installation, GGUF IS the model — the file is read directly during inference, not extracted.

Searching for "Mistral-7B-Instruct-v0.2-GGUF" on Hugging Face shows 12+ quantization variants of the same model. Q4_K_M is 4.37 GB (good for 8 GB VRAM). Q2_K is 3.08 GB (tight for 4 GB VRAM, with significant quality loss). You do NOT need to quantize yourself — pre-quantized files are available for virtually every popular model.

Sweet Spot

Q4 _K_M

Q4_K_M — recommended for most users

Caution

Common Misconception: I Need to Quantize Models Myself

No. Community providers and official model pages on Hugging Face offer pre-quantized GGUF files for download. You simply pick the right Q-level for your hardware and download. Some vendors even ship their models in 4 bits themselves and prepared them for it during training (quantization-aware training), such as Google’s Gemma 3 in its QAT versions or OpenAI’s gpt-oss. The quality loss is then smaller than with quantizing after the fact.

Why is it called Q4_K_M and not just Q4? The K stands for K-Quant, a method that stores the weights in small blocks, each with its own scaling factor. On top of that comes a mix: particularly sensitive weight matrices (such as parts of the attention and feed-forward weights) are stored at higher precision, while the rest are compressed more aggressively. The letters S, M, and L indicate how many weight matrices get this protection: L retains more information, S compresses more aggressively. Q4_K_M is the recommended compromise — significantly better than naive 4-bit quantization because the most important information is preserved.

Hardware Matching — Which Model Fits Your Machine?

Imagine loading furniture into a moving van. You cannot half-load a sofa — it either fits or it doesn't. Similarly, the model must fit entirely in memory. You can "compress" it (quantize) to make it fit, just as you might disassemble furniture. But a full living room set that barely fits (large model, heavy quantization) furnishes your apartment better than a single piece in mint condition (small model, no quantization).

The iron rule: your quantized model must fit completely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead for the OS and context, as long as the context stays short. Long contexts need more: for Llama 3.1 8B, the context memory (KV cache) takes about 1 GB per 8,000 tokens. The decision matrix maps VRAM to optimal model-quantization combinations. A key principle: within the same model family, a larger model at a lower quantization level (e.g., 13B-Q4) generally outperforms a smaller model at a higher quantization level (e.g., 8B-Q8) because broader parameter knowledge matters more than per-weight precision.

Unlike furniture, some parts of a model CAN be offloaded to slower RAM (GPU/CPU split), but this drastically reduces speed. Mixture-of-experts models such as Gemma 4 26B A4B are an exception: per token they compute with only a small share of their weights (3.8 of 25.2 billion). If you offload just their expert weights to RAM, they often remain quite usable.

4 GB VRAM 3B-Q4_K_M or 7B-Q2_K (tight) — very limited, simple tasks only
8 GB VRAM 7B-Q4_K_M (sweet spot) — solid performance on most tasks (approx. 4.4 GB + 1-2 GB overhead)
12 GB VRAM 13B-Q4_K_M or 7B-Q6_K — optimal range for local AI
24 GB VRAM models around 30B at Q4_K_M (e.g. Gemma 4 31B: 18.3 GB); 70B-Q4_K_M only with RAM offloading, slow — professional quality
8B-Q8 (higher precision)

8.5 GB. Higher precision per weight. Fits in 12 GB VRAM. Fewer parameters = narrower knowledge base.

13B-Q4_K_M (larger model)

7.9 GB. Lower precision per weight. Fits in 12 GB VRAM. More parameters = broader knowledge base. Within the same model family, generally better answers.

RTX 3060 with 12 GB VRAM. Option A: 8B-Q6_K = 6.6 GB (fits, high quality). Option B: 13B-Q4_K_M = 7.9 GB (fits, more knowledge). Option C: 70B-Q4_K_M = about 43 GB (does not fit, needs offloading — very slow). Recommendation: Option B, provided both models come from the same family and generation — then the larger model’s broader knowledge outweighs the smaller one’s higher per-weight precision. Be careful when mixing generations: in the same test by Meta, the older Llama 2 13B (2023) scores only 53.8% on the MMLU knowledge test, the newer Llama 3 8B (2024) 66.6%. Current families come in other sizes, for example Qwen3 with 8B and 14B: 14B-Q4_K_M weighs 9.0 GB and fits in 12 GB.

CPU-only: 7B-Q4_K_M in regular system memory runs at about 5 to 15 tokens per second, depending on memory speed — usable, but significantly slower than GPU inference.

Caution

Common Misconception: Always Use the Highest Quantization Level (Q8)

Counter-intuitive but true: with limited VRAM, a larger model from the same family at Q4 usually gives better answers than a smaller model at Q8. The total knowledge encoded in more parameters outweighs the precision advantage of more bits per weight.

Why does a 13B model at Q4 often outperform a 8B model at Q8? The answer lies in the nature of stored knowledge. Each parameter encodes relationships between words, concepts, and patterns. More parameters mean more relationships — a broader knowledge base. Reducing from Q8 to Q4 lowers the precision of individual weights, but the fundamental patterns remain recognisable. Benchmarks confirm this within a family: on the MMLU knowledge test, Llama 2 13B reaches 54.8%, Llama 2 7B only 45.3% (vendor figures at full precision). Q4_K_M costs little of that: for Llama 2, perplexity, a measure of prediction error, rises by only just over 1% according to llama.cpp measurements. The knowledge advantage of 6 billion additional parameters clearly outweighs the precision loss from heavier quantization.

Note

Summary

  • Precision determines model size: FP16 uses 2 bytes per weight, INT4 uses 0.5 bytes — a 4x reduction that shrinks a 14 GB model to 3.5 GB.
  • GGUF is the standard format for local models — one file contains everything. Q4_K_M is the recommended sweet spot for most users.
  • Always match model and quantization to your hardware: the model must fit entirely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead.

This article concludes the learning path “Local Language Models”. You now know where to find models, how to run them locally, and how to match them to your hardware. Ready for the next step?

Check

Quiz: Quantization

Not completed

What does "INT4 quantization" mean for a model's weights?

Select one answer

Answer Key: 1) B · 2) B · 3) B · 4) B · 5) B

Check

Knowledge Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done