The AI model fits on your computer. The memory problem is not over.

Quantisation represents model values with fewer bits, reducing the memory needed to store them. Hugging Face’s bitsandbytes documentation describes eight-bit inference and four-bit approaches, including methods that preserve higher precision for sensitive computations. It also exposes memory-footprint checks and options to distribute weights across devices.

The format is not simply a quality setting with one universal result. Supported hardware, model layers and the quantisation method all affect what runs. Offloading also needs careful interpretation: weights moved to CPU memory may use a different precision, and moving work away from the GPU changes the execution arrangement. A model file becoming small enough to download does not establish that its complete inference workload fits on a particular graphics card. Check the runtime’s actual allocation and evaluate the chosen precision on the tasks where small errors would matter, rather than assuming every compressed version preserves identical behaviour.

Ollama’s operational guidance separates model loading from the context and parallelism used to serve requests. Its process display can show whether a model is entirely on the GPU, in system memory or divided between them. It also allows a context setting to be changed. Increasing that allowance can consume additional working memory, even when the model’s stored weights are unchanged.

Parallel requests add another demand. The documentation explains that multiple requests increase the effective context allocation, and insufficient memory can cause requests to queue while other models unload. A machine that answers one short prompt comfortably may therefore struggle with several long agent sessions. Measure first-response time, generation speed and memory under the workload you actually intend to run. Then decide which trade-off is acceptable: a smaller model, shorter context, fewer simultaneous requests or some CPU offload. Local inference is a resource budget shared by the model and its work, rather than a one-time check of the download size.