Back to Learning Area

Running Models on Your Own PC (Local Inference)

Running AI on your own computer — if you have enough patience and enough RAM.

  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: AI on your own computer

What is local inference?

You can use AI programs on your computer. This is called local inference. Local inference means: the program runs on your computer. You do not need the internet. Your data stays with you.

Two tools

There are two famous tools. Ollama: you type commands into a black window. This is called a terminal. LM Studio: you click buttons on the screen. Both usually use the same program in the background. Ollama now also has a chat window. And LM Studio also works in the terminal.

What do I need?

Your computer needs enough working memory. Working memory is RAM. Small AI programs need 4 GB RAM. Medium ones need 8 GB. Large ones need 40 GB. A graphics card makes the program faster.

You have seen how many open models Hugging Face hosts: millions. But a model file sitting on a server is like an unplayed vinyl record — silent until you have a turntable. This article introduces two "turntables" (Ollama and LM Studio), the motor that spins them (llama.cpp), and how to check if your PC has enough power to play the records you want.

By the end, you'll know which tool fits you, what happens under the hood, and which models your machine can handle.

Ollama & LM Studio — Two Tools, One Purpose

Definition

Local LLM Inference

Analogy

Think of Ollama and LM Studio as two different music players for the same MP3 files. Ollama is like a command-line music player — you type "play song.mp3" and it works perfectly, but you need to know the commands. LM Studio is like a full graphical music player with album art, playlists, and an equalizer you can click. Both play the same music (run the same models), both can stream to other devices (API server), but they serve different user preferences.

Unlike music players which are truly equivalent, LM Studio offers some unique features Ollama lacks (side-by-side model comparison, visual RAM monitoring) and vice versa (Ollama's curated library is simpler to navigate). The line is blurring, though: since July 2025 Ollama has had a simple chat window for macOS and Windows, and since January 2026 LM Studio has had a chat in the terminal (lms chat).

Ollama — Terminal Workflow

  1. 1. Download model: ollama pull llama3.1:8b
  2. 2. Start chat: ollama run llama3.1:8b
  3. 3. Type a question and get a response

LM Studio — GUI Workflow

  1. 1. Search and download modelGGUF format, Q4_K_M quantization — compresses the model to fit on normal PCs
  2. 2. Configure parameters and set system prompt
  3. 3. Ask a question in the chat tab and get a response
Ollama

Terminal-based, since 2025 also with a simple chat window (macOS, Windows). One command to download, one to chat. Curated model library. Ideal for developers and terminal users.

LM Studio

Graphical interface. Hugging Face browser, model comparison, visual parameters. Ideal for beginners and visual users.

Note

Same Task — Two Approaches

Task: "Chat with Llama 3.1 8B in German." Ollama: Two commands in the terminal, after that a short answer arrives in ~3 seconds on an RTX 3060. LM Studio: Open app, search and download model, set system prompt, chat.

Both tools offer a local API server: Ollama starts it automatically on port 11434, LM Studio at the push of a button on a configurable port.

The key feature: the API is compatible with the OpenAI format. Any program, script, or plugin that can talk to ChatGPT also works with your local model — you just change the URL (for Ollama: http://localhost:11434/v1).

Example: API call with curl (Ollama's own interface)

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.1:8b",
    "prompt": "What is DNA?"
  }'

Both tools lower the barrier to local AI from "compile C++ code" to "run one command" or "click a button" — democratizing access to AI.

The Inference Engine — llama.cpp

Definition

Inference Engine

Analogy

If Ollama and LM Studio are the steering wheel and dashboard of a car, then llama.cpp is the engine under the hood. You turn the steering wheel (type a prompt), the engine does the actual mechanical work (mathematical computation), and the dashboard shows the result (generated text).

A car engine is a single physical component. llama.cpp is software that can leverage different hardware (CPU, GPU, or both) — it is more flexible than a physical engine.

What Happens During a Query?

  1. 1. Ollama/LM Studio passes the text to llama.cpp
  2. 2. llama.cpp tokenizes the text into numeric tokens
  3. 3. Tokens pass through the neural network layersmatrix multiplications
  4. 4. The next token is selected from the computed probabilities
  5. 5. Steps 3-4 repeat until the response is complete

llama.cpp was written by Georgi Gerganov in pure C/C++. This means: it runs on virtually any hardware, without Python dependencies. It is the invisible backbone that powers Ollama and LM Studio for most models. But llama.cpp no longer works alone: Ollama runs some models on its own engine built on the same base library (GGML), and on Apple Macs both Ollama and LM Studio also use Apple's MLX. Since February 2026, Gerganov and his team have been part of Hugging Face; llama.cpp remains open source.

llama.cpp is to local LLMs what the browser is to the web — the universal runtime that made local AI accessible to everyone.

Hardware Requirements — Can Your PC Handle It?

The amount of memory (RAM or VRAM) determines which model sizes you can run locally. Three execution modes exist:

GPU Inference Entire model in the GPU's VRAM. Fastest option: 20-100 tokens/second.
CPU Inference Model in regular RAM. Slower but universal: 5-15 tokens/second.
Hybrid Mode Model split between VRAM and RAM (GPU Offloading). Compromise between speed and model size.

Think of your computer's memory as a workbench. A small model (7B) is like a compact toolbox that fits on any desk. A medium model (13B) needs a proper workshop table. A huge model (70B) needs an industrial workbench that most homes don't have.

At a real workbench, it makes little difference whether a tool lies on the table or on the shelf. In computing, the speed difference between VRAM and RAM access is large (roughly 4 to 20 times, depending on the system), making placement critically important.

Model Size Rules of Thumb

7B Model (Q4)

~4  GB

Fits on almost any modern PC (~4-5 GB RAM)

13B Model (Q4)

~8  GB

Needs a decent workstation (~8 GB RAM)

70B Model (Q4)

~40  GB

Needs a high-end workstation (~40 GB RAM)

Three Hardware Scenarios

Laptop (16 GB RAM, no GPU)

Llama 3.1 8B Q4: ~5 GB in RAM, CPU inference, ~8 tokens/sec. Usable for chatting, with noticeable delay.

Gaming PC (RTX 3060, 12 GB VRAM)

Same model fits entirely in VRAM, GPU inference, ~40 tokens/sec. Text flows instantly.

Note

Workstation (RTX 4090, 24 GB VRAM)

Llama 3.1 70B Q4: just over 40 GB needed, Hybrid mode (24 GB in VRAM, around 20 GB in RAM), ~2-4 tokens/sec. Large model, usable but sluggish.

Tokens per second (tok/s) is the unit for generation speed. Human reading speed is about 4 words, or roughly 5 tokens, per second.

Generation feels fluid above ~10 tok/s. Below 5 tok/s, delays become noticeable. GPU inference typically achieves 20-100 tok/s, CPU inference 5-15 tok/s. One exception is mixture-of-experts models such as gpt-oss-20b: they need memory for all 21 billion parameters but compute each token with only 3.6 billion of them, so they run much faster than their size suggests.

Interactive: does the model fit your computer?

Now you can try the rules of thumb from above yourself. Pick a model size and a quantization. The calculator shows the memory needed and estimates for laptop, gaming PC and workstation whether the model runs smoothly, stutters or does not fit at all.

Caution

Warning: Three Common Myths

"Local models are as good as ChatGPT"

No. Local open models are getting better, but on a home PC they usually don't reach the quality of large cloud models like ChatGPT or Claude. They are good enough for many tasks, but not for all. The strongest open models often have a trillion parameters or more and won't fit on any home PC.

"You need an expensive graphics card"

No. CPU inference works with any modern PC. It's slower but perfectly usable. A 7B model runs on a laptop with 16 GB RAM.

"All local models are equally good"

No. Quality varies greatly depending on training method, data volume, and model architecture. A well-trained 7B model can be better than a poorly trained 13B model.

Note

Summary

  • Ollama (mainly terminal) and LM Studio (mainly GUI) are two paths to the same goal: running language models locally. Both offer an OpenAI-compatible API.
  • Beneath both tools works an inference engine (usually llama.cpp) that loads weights, processes tokens, and generates responses.
  • 7B Q4 needs ~4 GB (any PC), 13B Q4 needs ~8 GB, 70B Q4 needs ~40 GB. GPU is fast (20-100 tok/s), CPU is slower but universal (5-15 tok/s).

Check

Quiz: Local Inference

Not completed

Which tool provides a graphical model browser with Hugging Face integration and visual model comparison?

Select one answer

Answer Key: 1) C · 2) B · 3) B · 4) B · 5) B · 6) B

Check

Understanding Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done