Running Models on Your Own PC (Local Inference)
Running AI on your own computer — if you have enough patience and enough RAM.
You have seen how many open models Hugging Face hosts: millions. But a model file sitting on a server is like an unplayed vinyl record — silent until you have a turntable. This article introduces two "turntables" (Ollama and LM Studio), the motor that spins them (llama.cpp), and how to check if your PC has enough power to play the records you want.
By the end, you'll know which tool fits you, what happens under the hood, and which models your machine can handle.
Ollama & LM Studio — Two Tools, One Purpose
Definition
Local LLM Inference
Analogy
Definition
Unlike music players which are truly equivalent, LM Studio offers some unique features Ollama lacks (side-by-side model comparison, visual RAM monitoring) and vice versa (Ollama's curated library is simpler to navigate). The line is blurring, though: since July 2025 Ollama has had a simple chat window for macOS and Windows, and since January 2026 LM Studio has had a chat in the terminal (lms chat).
Ollama — Terminal Workflow
- 1. Download model: ollama pull llama3.1:8b
- 2. Start chat: ollama run llama3.1:8b
- 3. Type a question and get a response
LM Studio — GUI Workflow
- 1. Search and download modelGGUF format, Q4_K_M quantization — compresses the model to fit on normal PCs
- 2. Configure parameters and set system prompt
- 3. Ask a question in the chat tab and get a response
Terminal-based, since 2025 also with a simple chat window (macOS, Windows). One command to download, one to chat. Curated model library. Ideal for developers and terminal users.
Graphical interface. Hugging Face browser, model comparison, visual parameters. Ideal for beginners and visual users.
Note
Same Task — Two Approaches
Task: "Chat with Llama 3.1 8B in German." Ollama: Two commands in the terminal, after that a short answer arrives in ~3 seconds on an RTX 3060. LM Studio: Open app, search and download model, set system prompt, chat.
Both tools offer a local API server: Ollama starts it automatically on port 11434, LM Studio at the push of a button on a configurable port.
The key feature: the API is compatible with the OpenAI format. Any program, script, or plugin that can talk to ChatGPT also works with your local model — you just change the URL (for Ollama: http://localhost:11434/v1).
Example: API call with curl (Ollama's own interface)
curl http://localhost:11434/api/generate \
-d '{
"model": "llama3.1:8b",
"prompt": "What is DNA?"
}'Both tools lower the barrier to local AI from "compile C++ code" to "run one command" or "click a button" — democratizing access to AI.
The Inference Engine — llama.cpp
Definition
Inference Engine
Analogy
Definition
A car engine is a single physical component. llama.cpp is software that can leverage different hardware (CPU, GPU, or both) — it is more flexible than a physical engine.
What Happens During a Query?
- 1. Ollama/LM Studio passes the text to llama.cpp
- 2. llama.cpp tokenizes the text into numeric tokens
- 3. Tokens pass through the neural network layersmatrix multiplications
- 4. The next token is selected from the computed probabilities
- 5. Steps 3-4 repeat until the response is complete
llama.cpp was written by Georgi Gerganov in pure C/C++. This means: it runs on virtually any hardware, without Python dependencies. It is the invisible backbone that powers Ollama and LM Studio for most models. But llama.cpp no longer works alone: Ollama runs some models on its own engine built on the same base library (GGML), and on Apple Macs both Ollama and LM Studio also use Apple's MLX. Since February 2026, Gerganov and his team have been part of Hugging Face; llama.cpp remains open source.
llama.cpp is to local LLMs what the browser is to the web — the universal runtime that made local AI accessible to everyone.
Hardware Requirements — Can Your PC Handle It?
The amount of memory (RAM or VRAM) determines which model sizes you can run locally. Three execution modes exist:
Think of your computer's memory as a workbench. A small model (7B) is like a compact toolbox that fits on any desk. A medium model (13B) needs a proper workshop table. A huge model (70B) needs an industrial workbench that most homes don't have.
At a real workbench, it makes little difference whether a tool lies on the table or on the shelf. In computing, the speed difference between VRAM and RAM access is large (roughly 4 to 20 times, depending on the system), making placement critically important.
Model Size Rules of Thumb
7B Model (Q4)
~4 GB
Fits on almost any modern PC (~4-5 GB RAM)
13B Model (Q4)
~8 GB
Needs a decent workstation (~8 GB RAM)
70B Model (Q4)
~40 GB
Needs a high-end workstation (~40 GB RAM)
Three Hardware Scenarios
Llama 3.1 8B Q4: ~5 GB in RAM, CPU inference, ~8 tokens/sec. Usable for chatting, with noticeable delay.
Same model fits entirely in VRAM, GPU inference, ~40 tokens/sec. Text flows instantly.
Note
Workstation (RTX 4090, 24 GB VRAM)
Llama 3.1 70B Q4: just over 40 GB needed, Hybrid mode (24 GB in VRAM, around 20 GB in RAM), ~2-4 tokens/sec. Large model, usable but sluggish.
Tokens per second (tok/s) is the unit for generation speed. Human reading speed is about 4 words, or roughly 5 tokens, per second.
Generation feels fluid above ~10 tok/s. Below 5 tok/s, delays become noticeable. GPU inference typically achieves 20-100 tok/s, CPU inference 5-15 tok/s. One exception is mixture-of-experts models such as gpt-oss-20b: they need memory for all 21 billion parameters but compute each token with only 3.6 billion of them, so they run much faster than their size suggests.
Interactive: does the model fit your computer?
Now you can try the rules of thumb from above yourself. Pick a model size and a quantization. The calculator shows the memory needed and estimates for laptop, gaming PC and workstation whether the model runs smoothly, stutters or does not fit at all.
Caution
Warning: Three Common Myths
"Local models are as good as ChatGPT"
No. Local open models are getting better, but on a home PC they usually don't reach the quality of large cloud models like ChatGPT or Claude. They are good enough for many tasks, but not for all. The strongest open models often have a trillion parameters or more and won't fit on any home PC.
"You need an expensive graphics card"
No. CPU inference works with any modern PC. It's slower but perfectly usable. A 7B model runs on a laptop with 16 GB RAM.
"All local models are equally good"
No. Quality varies greatly depending on training method, data volume, and model architecture. A well-trained 7B model can be better than a poorly trained 13B model.
Note
Summary
- Ollama (mainly terminal) and LM Studio (mainly GUI) are two paths to the same goal: running language models locally. Both offer an OpenAI-compatible API.
- Beneath both tools works an inference engine (usually llama.cpp) that loads weights, processes tokens, and generates responses.
- 7B Q4 needs ~4 GB (any PC), 13B Q4 needs ~8 GB, 70B Q4 needs ~40 GB. GPU is fast (20-100 tok/s), CPU is slower but universal (5-15 tok/s).
Check
Quiz: Local Inference
Which tool provides a graphical model browser with Hugging Face integration and visual model comparison?
Select one answer
1. Which tool provides a graphical model browser with Hugging Face integration and visual model comparison?
2. You have a laptop with 16 GB RAM and no dedicated GPU. Which model size and inference mode should you choose?
3. Both Ollama and LM Studio offer an OpenAI-compatible API server. What practical advantage does this provide?
4. A friend claims: "I have an RTX 3060 with 12 GB VRAM. I can run Llama 70B at full GPU speed." What do you say?
5. Your friend wants to use a local LLM to summarize research papers. They have a Windows laptop with 8 GB RAM and no dedicated GPU. What do you recommend?
6. You're building a Python script that needs to query a local LLM. Both Ollama and LM Studio are installed. What shared feature of both tools makes this possible without writing custom inference code?
Check
Understanding Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done