Running Models on Your Own PC (Local Inference)

You have seen thousands of open-source models on Hugging Face. But a model file sitting on a server is like an unplayed vinyl record — silent until you have a turntable. This article introduces two "turntables" (Ollama and LM Studio), the motor that spins them (llama.cpp), and how to check if your PC has enough power to play the records you want.

By the end, you'll know which tool fits you, what happens under the hood, and which models your machine can handle.

Ollama & LM Studio — Two Tools, One Purpose

Unlike music players which are truly equivalent, LM Studio offers some unique features Ollama lacks (side-by-side model comparison, visual RAM monitoring) and vice versa (Ollama's curated library is simpler to navigate).

Ollama — Terminal Workflow

LM Studio — GUI Workflow

curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.1:8b",
    "prompt": "What is DNA?"
  }'

Both tools lower the barrier to local AI from "compile C++ code" to "run one command" or "click a button" — democratizing access to AI.

The Inference Engine — llama.cpp

A car engine is a single physical component. llama.cpp is software that can leverage different hardware (CPU, GPU, or both) — it is more flexible than a physical engine.

What Happens During a Query?

llama.cpp was written by Georgi Gerganov in pure C/C++. This means: it runs on virtually any hardware, without Python dependencies. It is the invisible backbone that powers Ollama and LM Studio for most models.

llama.cpp is to local LLMs what the browser is to the web — the universal runtime that made local AI accessible to everyone.

Hardware Requirements — Can Your PC Handle It?

The amount of memory (RAM or VRAM) determines which model sizes you can run locally. Three execution modes exist:

Think of your computer's memory as a workbench. A small model (7B) is like a compact toolbox that fits on any desk. A medium model (13B) needs a proper workshop table. A huge model (70B) needs an industrial workbench that most homes don't have.

At a real workbench, it makes little difference whether a tool lies on the table or on the shelf. In computing, the speed difference between VRAM and RAM access is enormous (10-100x), making placement critically important.

Model Size Rules of Thumb

Three Hardware Scenarios

Interactive: does the model fit your computer?

Now you can try the rules of thumb from above yourself. Pick a model size and a quantization. The calculator shows the memory needed and estimates for laptop, gaming PC and workstation whether the model runs smoothly, stutters or does not fit at all.