Two ways to use a language model
There are two broad ways to put a language model into a project. You can call a hosted service over the internet, sending your text and receiving a reply, or you can download the model’s parameters and run them yourself. Running locally means the work happens on hardware you control, such as a desktop PC, a home server or a single-board computer.
Neither option is simply better. The right choice depends on what you value: quality, cost, privacy, convenience, or working without a connection. The sections below describe the trade-offs in terms that stay useful even as specific models and tools change quickly.
Memory is the first constraint
A language model is a large set of numbers (parameters) that must be held in memory while it runs. As a rough rule, memory needed is about the number of parameters multiplied by the storage size of each one, plus extra for working data.
A worked example makes this concrete. Suppose a model has 7 billion parameters. If each is stored in 16 bits (2 bytes), the parameters alone take about 14 billion bytes, which is roughly 14 gigabytes. If each is stored in 8 bits (1 byte), it is about 7 gigabytes. If each is stored in 4 bits (half a byte), it is about 3.5 gigabytes. These figures cover only the parameters; the real requirement is higher once you add working memory, so treat them as lower bounds.
That arithmetic is why memory capacity, not raw processor speed, often decides what you can run. Where the model needs to live matters too: fast graphics card memory is usually limited, while ordinary system memory is more plentiful but slower to read from. The general hardware picture is covered under inference hardware.
Quantisation: smaller at a cost
Quantisation stores parameters with fewer bits, as in the example above. It lets larger models fit on modest machines and often speeds things up. The price is some loss of accuracy. How much depends on the model and the task, and lighter compression is often hard to notice while very aggressive compression tends to degrade quality more visibly. Try the specific model and compression level on your own tasks rather than relying on general claims.
Context length uses memory too
A model reads your prompt and its own previous output within a limited window called the context. Longer contexts need additional working memory, which grows as the conversation or document grows. If you plan to feed in long documents, budget for that on top of the parameters, and expect slower responses as the context fills.
Speed: what to expect
Speed is usually described as how quickly the model produces output tokens, which are small pieces of text. Generating each token requires reading through much of the model’s parameters, so speed is strongly tied to memory bandwidth. Larger models and heavier precision tend to be slower on the same hardware.
In general terms:
- A modern desktop CPU can run smaller or heavily compressed models, but slowly enough that you notice the wait.
- A graphics card with enough memory to hold the whole model is typically much faster.
- Single-board computers can run very small models, but for anything conversational expect a patient pace.
- When a model does not fit in graphics memory, tools may split it between the card and system memory, which usually works but slows generation noticeably.
Rather than trusting published numbers, measure on your own machine with prompts like the ones your project will use. Actual results vary with the software, settings and prompt length.
Quality: smaller models are not equal to the largest services
The biggest hosted models are generally more capable than what typical home hardware can hold. Models small enough to run locally can be useful for focused jobs such as summarising short text, extracting fields, classifying messages or handling a limited set of voice commands for a personal voice assistant. They are more likely to struggle with complex reasoning, long documents or broad knowledge. Their weaknesses include the habit of producing confident errors, so the habits discussed under model hallucinations and prompt design still apply, and smaller models often need simpler, more explicit prompts.
A reasonable approach is to start with the smallest model that does your task acceptably, and only move up if testing shows it is not enough.
Privacy and control
This is often the main reason to go local. With a local model, your text does not leave your machine, which avoids many of the concerns listed under cloud AI privacy. It also means no dependence on a provider’s availability, pricing changes or policy changes, and the project can work without internet access.
Be careful with the claim, though. Local does not automatically mean private or secure. Check what the software you run does: some tools collect usage data or fetch updates, and a local service exposed to your network can be reached by others if not configured carefully. Keep it off the public internet, use sensible access controls, and apply the same cautions as for any home server, including remote access arrangements and firmware updates for the host.
Cost
Local running trades per-use fees for upfront hardware and ongoing electricity. If you only make occasional requests, a hosted service may cost less than buying suitable hardware. If you make a great many requests, or the privacy value is high, local may be more sensible. Power draw matters for always-on machines, so use a proper battery runtime estimate or measure consumption if the device runs from batteries, and think about heat and ventilation for hardware running continuously.
Licences and model sources
Models are published under a variety of terms. Some permit broad use, and others limit commercial use or require attribution or acceptance of an agreement. Check the terms of the specific model before using it in anything you distribute. Treat the general ideas behind software licenses as a starting point, not a substitute for reading the actual model licence. Download models only from sources you trust, since files from unknown places are a security risk.
A simple way to choose
- Write down the task and how good the answers need to be.
- Decide how sensitive the data is and whether offline use matters.
- Estimate memory from parameter count and compression level, adding a safety margin.
- Try a small model locally and a hosted service side by side on the same sample prompts.
- Compare quality, speed, cost and privacy, then choose, or use a hybrid: local for sensitive or routine requests, hosted for hard ones.
Revisit the decision from time to time. Both hosted and local options change quickly, and what was impractical last year may be easy now.