Start with the workload, not the chip
Running a trained model is called inference, and the hardware that suits it depends on what you are running and how you are using it. A small model that classifies a sensor reading once a second has almost nothing in common with a large language model generating text for a chat window. Before comparing parts, write down four things:
- Model size: roughly how many parameters it has and what numeric precision they are stored in.
- Latency target: does an answer need to arrive in a fraction of a second, or is a few seconds fine?
- Throughput: one request at a time, or many at once?
- Power and space: is this plugged into a wall, running from a battery, or sealed in an enclosure?
If you are fuzzy on the difference between training and inference, the ideas behind model training explain why inference is usually the lighter job.
Why neural networks suit particular hardware
Most modern models spend the bulk of their time on large batches of multiplications and additions, often organised as matrix operations. These operations can be done in parallel: many small calculations at once, with few dependencies between them. Hardware that offers a lot of parallel arithmetic, and can feed data to it quickly, tends to do well.
Two ideas keep recurring:
- Compute: how many operations per second the chip can perform.
- Memory bandwidth and capacity: how fast data can be moved to the compute units, and how much can be held close by.
For many models, especially large language models, memory is the limiting factor more often than raw compute. The parameters must be read from memory for each output, so a chip with plenty of arithmetic but little memory can sit idle waiting for data, or may not be able to load the model at all.
Processors (CPUs)
A general-purpose CPU can run almost any model, which is its main strength. It is already in your computer or single-board computer, needs no special drivers beyond the software library you use, and handles the surrounding program logic as well. Modern CPUs include vector instructions that speed up the arithmetic.
The weakness is parallelism. A CPU has a modest number of powerful cores, so large models tend to run slowly. For small and medium models, occasional requests, or tasks where a one- or two-second delay is acceptable, a CPU is often enough, and it is the cheapest place to start. Many hobby projects on a Raspberry Pi hub work this way, running a compact model on the board’s own processor.
Graphics cards (GPUs)
A graphics card contains thousands of simpler cores built for parallel arithmetic, along with fast dedicated memory. This design happens to match neural network workloads well, which is why GPUs became the standard tool for both training and running models.
Things to weigh:
- Video memory (VRAM) is frequently the deciding number. If the model’s parameters do not fit, you must shrink the model or accept very slow workarounds.
- Software support differs between vendors and generations. Check that the libraries you plan to use support the card before buying.
- Power and heat: high-end cards draw significant power and need good cooling and a suitable supply, which affects the rest of a custom PC build or a small server.
- Cost: you pay for capability you may not use if your model is small.
A GPU is a good fit for larger models, for several simultaneous users, or when you want low latency. It is often overkill for a single small classifier.
Dedicated accelerators
An accelerator is a chip designed mainly for neural network arithmetic. You will see names such as neural processing unit (NPU), tensor processor, or vision accelerator. They come in several forms:
- Blocks built into a phone or laptop system-on-chip.
- Add-in modules for single-board computers, attached through a USB or expansion connector.
- Dedicated boards aimed at running vision or speech models at low power.
Their appeal is efficiency: more inference per watt than a CPU, often in a small package. The trade-offs are narrower flexibility and software that can be fussy. Accelerators typically support a defined set of operations and numeric formats, so a model may need to be converted or compressed before it runs, and some layers may not be supported at all. Always check the vendor documentation for the operations your model needs, and treat marketing figures for operations per second as a starting point, not a prediction of your results; the same caution applies to efficiency claims in any product sheet.
Shrinking the model to fit the hardware
Rather than buying bigger hardware, you can often make the model smaller. Common techniques include:
- Quantisation: storing parameters with fewer bits, which reduces memory and can speed things up, usually at some cost in accuracy that varies by model and task.
- Pruning: removing parameters that contribute little.
- Distillation: training a smaller model to imitate a larger one.
These are worth testing on your own data, because the accuracy loss on your task may be larger or smaller than on someone else’s. At the extreme end of this approach sits tiny machine learning, where models are squeezed to run on chips with kilobytes of memory.
Power, heat and the real environment
Specifications describe peak behaviour. In a real enclosure, sustained load raises temperature and a chip may reduce its speed to protect itself. If the device runs from batteries, estimate runtime from capacity and load using a battery runtime estimate rather than guessing, and remember that inference that runs continuously costs far more energy than inference triggered occasionally. Safe power supply and cooling practices apply to any build; follow manufacturer guidance.
Running the model close to where data is produced, rather than sending it elsewhere, is the idea behind edge computing. It can cut delay and bandwidth, but it moves the hardware decision onto you.
A simple decision path
- Pick or estimate the model and note its memory requirement.
- Test it on the CPU you already have. Measure real latency on realistic input.
- If too slow, try a smaller or quantised version before buying anything.
- If still too slow or too large, consider a GPU for flexibility, or an accelerator for low-power fixed tasks.
- Re-check software support and power budget before purchasing.
Whether to host a model yourself at all, or call an online service instead, is a separate question with its own trade-offs, covered under local language models.