Shrinking models for home use — the technique that makes local AI practical in the first place.
Concepts 10 min Intermediate June 22, 2026
The latest open-source language models are incredibly capable — but they weigh 14 GB or more. Your gaming laptop might have 8 GB of VRAM. Does that mean local AI is out of reach?
Not at all. Quantization compresses these models by reducing the precision of their billions of weights — shrinking them by up to 80% while keeping most of their intelligence intact. This article shows you exactly how it works and which model fits your machine.
Precision — How Many Bits Per Number?
Quantization
AnalogyDefinition
Think of the number Pi. At full precision (FP32), you store "3.141592653589" — perfect for calculations. At half precision (FP16), you store "3.14159" — good enough for almost everything. At 4-bit (INT4), you store just "3" — a rough approximation, but still correct enough to know that a circle with diameter 1 has a circumference of about 3.
Analogy:
Think of the number Pi. At full precision (FP32), you store "3.141592653589" — perfect for calculations. At half precision (FP16), you store "3.14159" — good enough for almost everything. At 4-bit (INT4), you store just "3" — a rough approximation, but still correct enough to know that a circle with diameter 1 has a circumference of about 3.
Definition:
Quantization reduces the numerical precision of model weights to decrease size and increase speed. Precision defines how many bits the computer uses to store each weight in a neural network. Higher precision (FP32 = 32 bits) means more accurate numbers but more memory. Lower precision (INT4 = 4 bits) means rougher approximations but drastically less memory.
Unlike rounding a single number, quantizing a neural network affects billions of weights simultaneously. The interactions between rounded weights can compound errors unpredictably. However, in practice, modern quantization methods distribute errors so well that quality loss stays minimal.
Llama 3.1 8B: FP16 = 16 GB, Q8_0 = 8.5 GB, Q4_K_M = 4.9 GB, Q2_K = 3.2 GB. From FP16 to Q4_K_M, the model shrinks by ~70%. Measured quality loss on benchmarks: under 5% for most tasks.
~70%
Size Reduction ~70% smaller from FP16 to Q4_K_M
<5%
Quality Loss Under 5% for Q4_K_M on most tasks
Common Misconception: Quantization Destroys Model Quality
For INT4 (Q4_K_M), quality loss is typically under 5% on standard benchmarks. For Q8_0, it is nearly indistinguishable from FP16. Only at Q2 does degradation become noticeable — but Q4 and above preserve the vast majority of model capability.
Interactive: What Does Each Precision Level Cost?
The article explained the four precision levels FP32, FP16, INT8, and INT4. Here you see them as an interactive comparison matrix: click a row and observe how size, speed, and quality behave at each level. The heatmap colors instantly show you where the advantages and trade-offs lie.
Precision Matrix: Quantization Levels Compared
Advantage (small/fast/high)
Neutral
Disadvantage (large/slow/low)
Click a row for detail view
Format
Bits/Param
Model Size
Inference Speed
Quality
VRAM (7B)
▸FP32
32
100 %
1x
100 %
~28 GB
▸FP16
16
50 %
2x
99 %
~14 GB
▸INT8
8
25 %
3x
95 %
~7 GB
▸INT4
4
12.5 %
5x
90 %
~3.5 GB
Key Takeaway: Quantization is a trade-off. INT4 models need only 12.5% of the memory and run up to 5x faster — with only 5-10% quality loss. For many applications, that's an excellent deal.
GGUF — One File, Everything Included
GGUF is like a self-contained app installer — as opposed to the old approach of manually downloading separate parts (weights, config, tokenizer) and assembling them yourself. One file, double-click, done.
GGUF (GGML Universal File) is the standard file format for running local language models, developed for llama.cpp. It is monolithic: one file contains model weights, tokenizer, configuration, and metadata. The naming convention encodes everything: model-params-quant.gguf. Q-levels (Q2_K through Q8_0) indicate the quantization level. The K suffix means K-Quant (mixed-precision method), S/M/L indicates how many weights are stored at higher precision.
Unlike an app installer which deletes itself after installation, GGUF IS the model — the file is read directly during inference, not extracted.
Searching for "Mistral-7B-Instruct-v0.2-GGUF" on Hugging Face shows 12+ quantization variants of the same model. Q4_K_M is 4.37 GB (good for 8 GB VRAM). Q2_K is 3.08 GB (for 4 GB VRAM). You do NOT need to quantize yourself — pre-quantized files are available for virtually every popular model.
Q4_K_M
Sweet Spot Q4_K_M — recommended for most users
Common Misconception: I Need to Quantize Models Myself
No. Community providers and official model pages on Hugging Face offer pre-quantized GGUF files for download. You simply pick the right Q-level for your hardware and download.
K-Quant: Mixed Precision
Why is it called Q4_K_M and not just Q4? The K stands for K-Quant, a method that does not treat all layers equally. Critical layers (such as attention layers) are stored at higher precision, while less important layers are compressed more aggressively. The letters S, M, and L indicate how many layers get this protection: L retains more information, S compresses more aggressively. Q4_K_M is the recommended compromise — significantly better than naive 4-bit quantization because the most important information is preserved.
Hardware Matching — Which Model Fits Your Machine?
Imagine loading furniture into a moving van. You cannot half-load a sofa — it either fits or it doesn't. Similarly, the model must fit entirely in memory. You can "compress" it (quantize) to make it fit, just as you might disassemble furniture. But a full living room set that barely fits (large model, heavy quantization) furnishes your apartment better than a single piece in mint condition (small model, no quantization).
The iron rule: your quantized model must fit completely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead for the OS and context. The decision matrix maps VRAM to optimal model-quantization combinations. A key principle: within the same model family, a larger model at a lower quantization level (e.g., 13B-Q4) generally outperforms a smaller model at a higher quantization level (e.g., 8B-Q8) because broader parameter knowledge matters more than per-weight precision.
Unlike furniture, some parts of a model CAN be offloaded to slower RAM (GPU/CPU split), but this drastically reduces speed.
4 GB VRAM 3B-Q4_K_M or 7B-Q2_K (tight) — very limited, simple tasks only
8 GB VRAM 7B-Q4_K_M (sweet spot) — solid performance on most tasks (approx. 4 GB + 1-2 GB overhead)
12 GB VRAM 13B-Q4_K_M or 7B-Q6_K — optimal range for local AI
24 GB VRAM 70B-Q4_K_M (only with RAM offloading, slow) or 13B-Q8_0 — professional quality
8B-Q8 (higher precision)
8.5 GB. Higher precision per weight. Fits in 12 GB VRAM. Fewer parameters = narrower knowledge base.
13B-Q4_K_M (larger model)
7.9 GB. Lower precision per weight. Fits in 12 GB VRAM. More parameters = broader knowledge base. Generally better answers.
RTX 3060 with 12 GB VRAM. Option A: 8B-Q6_K = 6.6 GB (fits, high quality). Option B: 13B-Q4_K_M = 7.9 GB (fits, more knowledge). Option C: 70B-Q4_K_M = 40 GB (does not fit, needs offloading — very slow). Recommendation: Option B — the 13B model's broader knowledge outweighs the 8B's higher per-weight precision.
CPU-only: 7B-Q4_K_M on 32 GB RAM runs at ~8 tokens per second — usable, but significantly slower than GPU inference.
Common Misconception: Always Use the Highest Quantization Level (Q8)
Counter-intuitive but true: with limited VRAM, a larger model at Q4 usually gives better answers than a smaller model at Q8. The total knowledge encoded in more parameters outweighs the precision advantage of more bits per weight.
Larger Model, Lower Quantization — Why?
Why does a 13B model at Q4 often outperform a 8B model at Q8? The answer lies in the nature of stored knowledge. Each parameter encodes relationships between words, concepts, and patterns. More parameters mean more relationships — a broader knowledge base. Reducing from Q8 to Q4 lowers the precision of individual weights, but the fundamental patterns remain recognisable. Benchmarks confirm this: Llama 2 13B-Q4_K_M scores better on MMLU (~55%) than Llama 2 7B-Q8_0 (~46%). The knowledge advantage of 6 billion additional parameters clearly outweighs the precision loss from heavier quantization.
Summary
Precision determines model size: FP16 uses 2 bytes per weight, INT4 uses 0.5 bytes — a 4x reduction that shrinks a 14 GB model to 3.5 GB.
GGUF is the standard format for local models — one file contains everything. Q4_K_M is the recommended sweet spot for most users.
Always match model and quantization to your hardware: the model must fit entirely in VRAM (GPU) or RAM (CPU), plus 1-2 GB overhead.
This article concludes the learning path “Local Language Models”. You now know where to find models, how to run them locally, and how to match them to your hardware. Ready for the next step?
Quiz: Quantization
Question 1 / 5
Not completed
What does "INT4 quantization" mean for a model's weights?
1. What does "INT4 quantization" mean for a model's weights?
☐ A) Each weight is stored using 4 bytes (32 bits)
☐ B) Each weight is stored using 4 bits (0.5 bytes)
☐ C) The model has exactly 4 billion parameters
☐ D) The model is split into 4 separate files
2. A model has 7 billion parameters. Approximately how much memory does it need in FP16 vs INT4?
☐ A) FP16 = 7 GB, INT4 = 1.75 GB
☐ B) FP16 = 14 GB, INT4 = 3.5 GB
☐ C) FP16 = 28 GB, INT4 = 7 GB
☐ D) FP16 = 3.5 GB, INT4 = 14 GB
3. You see the filename mistral-7b-instruct-Q4_K_M.gguf. What does "Q4_K_M" tell you?
☐ A) The model was trained for 4 epochs using the Keras framework, Medium batch size
☐ B) 4-bit quantization using the K-Quant mixed-precision method, Medium variant (M)
☐ C) The model has 4,000 layers optimized for mobile devices
☐ D) Quarter-precision with 4 KB context window
4. You have 12 GB VRAM. Which setup would you choose and why?
☐ A) 8B-Q6_K (6.6 GB) — highest quality that safely fits
☐ B) 13B-Q4_K_M (7.9 GB) — larger model, still fits, broader knowledge
☐ C) 70B-Q2_K (24 GB) — the biggest model is always best
☐ D) 7B-FP16 (14 GB) — no quality loss is worth the extra cost
5. A friend claims: "I always download Q8 files because higher quantization means better answers." You have a GPU with 12 GB VRAM. Why might this strategy backfire?
☐ A) Q8 files are often corrupted and crash during inference
☐ B) At 12 GB VRAM, Q8 limits you to smaller models (e.g., 8B). A 13B model at Q4_K_M would fit instead, and the broader knowledge of 13B parameters likely outperforms the higher per-weight precision of 8B-Q8
☐ C) Q8 files only work on NVIDIA GPUs, not on AMD
☐ D) Higher quantization numbers mean more compression, so Q8 is actually lower quality than Q4
Answer Key: 1) B · 2) B · 3) B · 4) B · 5) B
Knowledge Check
When you round a number to fewer decimal places, you lose precision. Why does an AI model still work surprisingly well when billions of weights are reduced to just 4 bits (Q4)?
You find a file called llama-3-8b-Q4_K_M.gguf on Hugging Face. What do the individual parts of the name tell you about the model?
You have a graphics card with 12 GB of memory. Would you choose an 8B model at Q8 or a 13B model at Q4? Explain your reasoning.