Quantization — Shrinking Models
Shrinking models for home use — the technique that makes local AI practical in the first place.
The latest open-source language models are incredibly capable — but they weigh 14 GB or more. Your gaming laptop might have 8 GB of VRAM. Does that mean local AI is out of reach?
Not at all. Quantization compresses these models by reducing the precision of their billions of weights — shrinking them by up to 80% while keeping most of their intelligence intact. This article shows you exactly how it works and which model fits your machine.
Precision — How Many Bits Per Number?
Definition
Quantization
Analogy
Definition
Unlike rounding a single number, quantizing a neural network affects billions of weights simultaneously. The interactions between rounded weights can compound errors unpredictably. However, in practice, modern quantization methods distribute errors so well that quality loss stays minimal.
Llama 3.1 8B: FP16 = 16 GB, Q8_0 = 8.5 GB, Q4_K_M = 4.9 GB, Q2_K = 3.2 GB. Why 4.9 GB and not the calculated 4 GB? Q4_K_M uses about 4.9 bits per weight on average, because scaling factors and a few weight matrices kept at higher precision are stored as well. From FP16 to Q4_K_M, the model shrinks by ~70%. Measured quality loss on benchmarks: under 5% for most tasks.
Size Reduction
~70 %
~70% smaller from FP16 to Q4_K_M
Quality Loss
<5 %
Under 5% for Q4_K_M on most tasks
Caution
Common Misconception: Quantization Destroys Model Quality
At 4 bits (Q4_K_M), quality loss is typically under 5% on standard benchmarks. For Q8_0, it is nearly indistinguishable from FP16. Below Q4 the loss rises quickly: at Q3 it becomes noticeable, at Q2 substantial — but Q4 and above preserve the vast majority of model capability.
Interactive: What Does Each Precision Level Cost?
The article explained the four precision levels FP32, FP16, INT8, and INT4. Here you see them as an interactive comparison matrix: click a row and observe how size, speed, and quality behave at each level. The heatmap colors instantly show you where the advantages and trade-offs lie.
Precision Matrix: Quantization Levels Compared
Click a row for detail view
| Format | Bits/Param | Model Size | Inference Speed | Quality | VRAM (7B) |
|---|---|---|---|---|---|
| FP32 | 32 | 100 % | 1x | 100 % | ~28 GB |
| FP16 | 16 | 50 % | 2x | 99 % | ~14 GB |
| INT8 | 8 | 25 % | 3x | 99 % | ~7 GB |
| INT4 | 4 | 12.5 % | 5x | 96 % | ~3.5 GB |
GGUF — One File, Everything Included
GGUF is like a self-contained app installer — as opposed to the old approach of manually downloading separate parts (weights, config, tokenizer) and assembling them yourself. One file, double-click, done.
GGUF (the successor to the older GGML format) is the standard file format for running local language models, developed for llama.cpp. It is monolithic: one file contains model weights, tokenizer, configuration, and metadata. The naming convention encodes everything: model-params-quant.gguf. Q-levels (Q2_K through Q8_0) indicate the quantization level. The K suffix means K-Quant (block-wise quantization, partly with mixed precision), S/M/L indicates how many weights are stored at higher precision. llama.cpp also knows further levels, such as the IQ variants and 4-bit floating-point formats like MXFP4 (used by gpt-oss) and, since March 2026, NVIDIA’s NVFP4.
Unlike an app installer which deletes itself after installation, GGUF IS the model — the file is read directly during inference, not extracted.
Searching for "Mistral-7B-Instruct-v0.2-GGUF" on Hugging Face shows 12+ quantization variants of the same model. Q4_K_M is 4.37 GB (good for 8 GB VRAM). Q2_K is 3.08 GB (tight for 4 GB VRAM, with significant quality loss). You do NOT need to quantize yourself — pre-quantized files are available for virtually every popular model.
Sweet Spot
Q4 _K_M
Q4_K_M — recommended for most users
Caution
Common Misconception: I Need to Quantize Models Myself
No. Community providers and official model pages on Hugging Face offer pre-quantized GGUF files for download. You simply pick the right Q-level for your hardware and download. Some vendors even ship their models in 4 bits themselves and prepared them for it during training (quantization-aware training), such as Google’s Gemma 3 in its QAT versions or OpenAI’s gpt-oss. The quality loss is then smaller than with quantizing after the fact.
Why is it called Q4_K_M and not just Q4? The K stands for K-Quant, a method that stores the weights in small blocks, each with its own scaling factor. On top of that comes a mix: particularly sensitive weight matrices (such as parts of the attention and feed-forward weights) are stored at higher precision, while the rest are compressed more aggressively. The letters S, M, and L indicate how many weight matrices get this protection: L retains more information, S compresses more aggressively. Q4_K_M is the recommended compromise — significantly better than naive 4-bit quantization because the most important information is preserved.
Hardware Matching — Which Model Fits Your Machine?
Imagine loading furniture into a moving van. You cannot half-load a sofa — it either fits or it doesn't. Similarly, the model must fit entirely in memory. You can "compress" it (quantize) to make it fit, just as you might disassemble furniture. But a full living room set that barely fits (large model, heavy quantization) furnishes your apartment better than a single piece in mint condition (small model, no quantization).
The iron rule: your quantized model must fit completely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead for the OS and context, as long as the context stays short. Long contexts need more: for Llama 3.1 8B, the context memory (KV cache) takes about 1 GB per 8,000 tokens. The decision matrix maps VRAM to optimal model-quantization combinations. A key principle: within the same model family, a larger model at a lower quantization level (e.g., 13B-Q4) generally outperforms a smaller model at a higher quantization level (e.g., 8B-Q8) because broader parameter knowledge matters more than per-weight precision.
Unlike furniture, some parts of a model CAN be offloaded to slower RAM (GPU/CPU split), but this drastically reduces speed. Mixture-of-experts models such as Gemma 4 26B A4B are an exception: per token they compute with only a small share of their weights (3.8 of 25.2 billion). If you offload just their expert weights to RAM, they often remain quite usable.
8.5 GB. Higher precision per weight. Fits in 12 GB VRAM. Fewer parameters = narrower knowledge base.
7.9 GB. Lower precision per weight. Fits in 12 GB VRAM. More parameters = broader knowledge base. Within the same model family, generally better answers.
RTX 3060 with 12 GB VRAM. Option A: 8B-Q6_K = 6.6 GB (fits, high quality). Option B: 13B-Q4_K_M = 7.9 GB (fits, more knowledge). Option C: 70B-Q4_K_M = about 43 GB (does not fit, needs offloading — very slow). Recommendation: Option B, provided both models come from the same family and generation — then the larger model’s broader knowledge outweighs the smaller one’s higher per-weight precision. Be careful when mixing generations: in the same test by Meta, the older Llama 2 13B (2023) scores only 53.8% on the MMLU knowledge test, the newer Llama 3 8B (2024) 66.6%. Current families come in other sizes, for example Qwen3 with 8B and 14B: 14B-Q4_K_M weighs 9.0 GB and fits in 12 GB.
CPU-only: 7B-Q4_K_M in regular system memory runs at about 5 to 15 tokens per second, depending on memory speed — usable, but significantly slower than GPU inference.
Caution
Common Misconception: Always Use the Highest Quantization Level (Q8)
Counter-intuitive but true: with limited VRAM, a larger model from the same family at Q4 usually gives better answers than a smaller model at Q8. The total knowledge encoded in more parameters outweighs the precision advantage of more bits per weight.
Why does a 13B model at Q4 often outperform a 8B model at Q8? The answer lies in the nature of stored knowledge. Each parameter encodes relationships between words, concepts, and patterns. More parameters mean more relationships — a broader knowledge base. Reducing from Q8 to Q4 lowers the precision of individual weights, but the fundamental patterns remain recognisable. Benchmarks confirm this within a family: on the MMLU knowledge test, Llama 2 13B reaches 54.8%, Llama 2 7B only 45.3% (vendor figures at full precision). Q4_K_M costs little of that: for Llama 2, perplexity, a measure of prediction error, rises by only just over 1% according to llama.cpp measurements. The knowledge advantage of 6 billion additional parameters clearly outweighs the precision loss from heavier quantization.
Note
Summary
- Precision determines model size: FP16 uses 2 bytes per weight, INT4 uses 0.5 bytes — a 4x reduction that shrinks a 14 GB model to 3.5 GB.
- GGUF is the standard format for local models — one file contains everything. Q4_K_M is the recommended sweet spot for most users.
- Always match model and quantization to your hardware: the model must fit entirely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead.
This article concludes the learning path “Local Language Models”. You now know where to find models, how to run them locally, and how to match them to your hardware. Ready for the next step?
Check
Quiz: Quantization
What does "INT4 quantization" mean for a model's weights?
Select one answer
1. What does "INT4 quantization" mean for a model's weights?
2. A model has 7 billion parameters. Approximately how much memory does it need in FP16 vs INT4?
3. You see the filename mistral-7b-instruct-Q4_K_M.gguf. What does "Q4_K_M" tell you?
4. You have 12 GB VRAM. Which setup would you choose and why?
5. A friend claims: "I always download Q8 files because higher quantization means better answers." You have a GPU with 12 GB VRAM. Why might this strategy backfire?
Check
Knowledge Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done